Aug 04 17:59:46.293776 ip-10-0-135-124 systemd[1]: Starting Kubernetes Kubelet...
Aug 04 17:59:46.746785 ip-10-0-135-124 kubenswrapper[2583]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Aug 04 17:59:46.746785 ip-10-0-135-124 kubenswrapper[2583]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Aug 04 17:59:46.746785 ip-10-0-135-124 kubenswrapper[2583]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Aug 04 17:59:46.746785 ip-10-0-135-124 kubenswrapper[2583]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Aug 04 17:59:46.746785 ip-10-0-135-124 kubenswrapper[2583]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Aug 04 17:59:46.747924 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.747837    2583 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Aug 04 17:59:46.752914 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.752898    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Aug 04 17:59:46.752914 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.752914    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Aug 04 17:59:46.752976 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.752918    2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Aug 04 17:59:46.752976 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.752922    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Aug 04 17:59:46.752976 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.752925    2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Aug 04 17:59:46.752976 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.752928    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Aug 04 17:59:46.752976 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.752931    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Aug 04 17:59:46.752976 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.752934    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Aug 04 17:59:46.752976 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.752937    2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Aug 04 17:59:46.752976 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.752939    2583 feature_gate.go:328] unrecognized feature gate: SignatureStores
Aug 04 17:59:46.752976 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.752942    2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Aug 04 17:59:46.752976 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.752947    2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Aug 04 17:59:46.752976 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.752950    2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Aug 04 17:59:46.752976 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.752954    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Aug 04 17:59:46.752976 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.752956    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Aug 04 17:59:46.752976 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.752959    2583 feature_gate.go:328] unrecognized feature gate: DualReplica
Aug 04 17:59:46.752976 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.752962    2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Aug 04 17:59:46.752976 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.752964    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Aug 04 17:59:46.752976 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.752967    2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Aug 04 17:59:46.752976 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.752970    2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Aug 04 17:59:46.752976 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.752972    2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Aug 04 17:59:46.753437 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.752975    2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Aug 04 17:59:46.753437 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.752978    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Aug 04 17:59:46.753437 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.752981    2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Aug 04 17:59:46.753437 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.752986    2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Aug 04 17:59:46.753437 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.752999    2583 feature_gate.go:328] unrecognized feature gate: NewOLM
Aug 04 17:59:46.753437 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.753002    2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Aug 04 17:59:46.753437 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.753005    2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Aug 04 17:59:46.753437 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.753008    2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Aug 04 17:59:46.753437 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.753012    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Aug 04 17:59:46.753437 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.753014    2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Aug 04 17:59:46.753437 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.753017    2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Aug 04 17:59:46.753437 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.753019    2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Aug 04 17:59:46.753437 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.753022    2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Aug 04 17:59:46.753437 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.753026    2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Aug 04 17:59:46.753437 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.753030    2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Aug 04 17:59:46.753437 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.753032    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Aug 04 17:59:46.753437 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.753035    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Aug 04 17:59:46.753437 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.753038    2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Aug 04 17:59:46.753437 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.753041    2583 feature_gate.go:328] unrecognized feature gate: OVNObservability
Aug 04 17:59:46.753437 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.753043    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Aug 04 17:59:46.753933 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.753046    2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Aug 04 17:59:46.753933 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.753049    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Aug 04 17:59:46.753933 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.753051    2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Aug 04 17:59:46.753933 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.753054    2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Aug 04 17:59:46.753933 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.753056    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Aug 04 17:59:46.753933 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.753059    2583 feature_gate.go:328] unrecognized feature gate: EventTTL
Aug 04 17:59:46.753933 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.753061    2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Aug 04 17:59:46.753933 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.753064    2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Aug 04 17:59:46.753933 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.753067    2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Aug 04 17:59:46.753933 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.753069    2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Aug 04 17:59:46.753933 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.753072    2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Aug 04 17:59:46.753933 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.753076    2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Aug 04 17:59:46.753933 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.753078    2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Aug 04 17:59:46.753933 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.753082    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Aug 04 17:59:46.753933 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.753085    2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Aug 04 17:59:46.753933 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.753087    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Aug 04 17:59:46.753933 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.753090    2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Aug 04 17:59:46.753933 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.753092    2583 feature_gate.go:328] unrecognized feature gate: Example
Aug 04 17:59:46.753933 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.753095    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Aug 04 17:59:46.754430 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.753099    2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Aug 04 17:59:46.754430 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.753103    2583 feature_gate.go:328] unrecognized feature gate: OSStreams
Aug 04 17:59:46.754430 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.753105    2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Aug 04 17:59:46.754430 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.753108    2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Aug 04 17:59:46.754430 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.753121    2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Aug 04 17:59:46.754430 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.753124    2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Aug 04 17:59:46.754430 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.753127    2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Aug 04 17:59:46.754430 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.753130    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Aug 04 17:59:46.754430 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.753133    2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Aug 04 17:59:46.754430 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.753135    2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Aug 04 17:59:46.754430 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.753138    2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Aug 04 17:59:46.754430 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.753141    2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Aug 04 17:59:46.754430 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.753143    2583 feature_gate.go:328] unrecognized feature gate: Example2
Aug 04 17:59:46.754430 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.753146    2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Aug 04 17:59:46.754430 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.753148    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Aug 04 17:59:46.754430 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.753151    2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Aug 04 17:59:46.754430 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.753154    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Aug 04 17:59:46.754430 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.753157    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Aug 04 17:59:46.754430 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.753159    2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Aug 04 17:59:46.754899 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.753162    2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Aug 04 17:59:46.754899 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.753165    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Aug 04 17:59:46.754899 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.753169    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Aug 04 17:59:46.754899 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.753172    2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Aug 04 17:59:46.754899 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.753174    2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Aug 04 17:59:46.754899 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.753177    2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Aug 04 17:59:46.754899 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.753180    2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Aug 04 17:59:46.754899 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.753182    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Aug 04 17:59:46.754899 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.753185    2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Aug 04 17:59:46.754899 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.753188    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Aug 04 17:59:46.754899 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.753190    2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Aug 04 17:59:46.754899 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.753193    2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Aug 04 17:59:46.754899 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.753196    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Aug 04 17:59:46.754899 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.753198    2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Aug 04 17:59:46.754899 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.753201    2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Aug 04 17:59:46.754899 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.753204    2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Aug 04 17:59:46.754899 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.753206    2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Aug 04 17:59:46.754899 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.753209    2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Aug 04 17:59:46.754899 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.753212    2583 feature_gate.go:328] unrecognized feature gate: PinnedImages
Aug 04 17:59:46.754899 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.753214    2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Aug 04 17:59:46.755384 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.753939    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Aug 04 17:59:46.755384 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.753946    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Aug 04 17:59:46.755384 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.753950    2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Aug 04 17:59:46.755384 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.753953    2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Aug 04 17:59:46.755384 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.753956    2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Aug 04 17:59:46.755384 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.753958    2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Aug 04 17:59:46.755384 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.753961    2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Aug 04 17:59:46.755384 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.753964    2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Aug 04 17:59:46.755384 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.753966    2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Aug 04 17:59:46.755384 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.753969    2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Aug 04 17:59:46.755384 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.753972    2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Aug 04 17:59:46.755384 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.753975    2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Aug 04 17:59:46.755384 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.753978    2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Aug 04 17:59:46.755384 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.753981    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Aug 04 17:59:46.755384 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.753984    2583 feature_gate.go:328] unrecognized feature gate: Example2
Aug 04 17:59:46.755384 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.753987    2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Aug 04 17:59:46.755384 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.753990    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Aug 04 17:59:46.755384 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.753992    2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Aug 04 17:59:46.755384 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.753995    2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Aug 04 17:59:46.755384 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.753998    2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Aug 04 17:59:46.755890 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.754001    2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Aug 04 17:59:46.755890 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.754004    2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Aug 04 17:59:46.755890 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.754007    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Aug 04 17:59:46.755890 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.754011    2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Aug 04 17:59:46.755890 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.754014    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Aug 04 17:59:46.755890 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.754017    2583 feature_gate.go:328] unrecognized feature gate: EventTTL
Aug 04 17:59:46.755890 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.754020    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Aug 04 17:59:46.755890 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.754023    2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Aug 04 17:59:46.755890 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.754026    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Aug 04 17:59:46.755890 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.754029    2583 feature_gate.go:328] unrecognized feature gate: DualReplica
Aug 04 17:59:46.755890 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.754032    2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Aug 04 17:59:46.755890 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.754034    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Aug 04 17:59:46.755890 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.754037    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Aug 04 17:59:46.755890 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.754039    2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Aug 04 17:59:46.755890 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.754042    2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Aug 04 17:59:46.755890 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.754044    2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Aug 04 17:59:46.755890 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.754047    2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Aug 04 17:59:46.755890 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.754050    2583 feature_gate.go:328] unrecognized feature gate: OVNObservability
Aug 04 17:59:46.755890 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.754052    2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Aug 04 17:59:46.756351 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.754055    2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Aug 04 17:59:46.756351 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.754057    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Aug 04 17:59:46.756351 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.754059    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Aug 04 17:59:46.756351 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.754064    2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Aug 04 17:59:46.756351 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.754068    2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Aug 04 17:59:46.756351 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.754071    2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Aug 04 17:59:46.756351 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.754074    2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Aug 04 17:59:46.756351 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.754077    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Aug 04 17:59:46.756351 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.754080    2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Aug 04 17:59:46.756351 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.754082    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Aug 04 17:59:46.756351 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.754085    2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Aug 04 17:59:46.756351 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.754088    2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Aug 04 17:59:46.756351 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.754092    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Aug 04 17:59:46.756351 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.754096    2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Aug 04 17:59:46.756351 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.754099    2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Aug 04 17:59:46.756351 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.754102    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Aug 04 17:59:46.756351 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.754105    2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Aug 04 17:59:46.756351 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.754107    2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Aug 04 17:59:46.756351 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.754110    2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Aug 04 17:59:46.756837 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.754113    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Aug 04 17:59:46.756837 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.754115    2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Aug 04 17:59:46.756837 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.754118    2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Aug 04 17:59:46.756837 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.754121    2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Aug 04 17:59:46.756837 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.754123    2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Aug 04 17:59:46.756837 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.754126    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Aug 04 17:59:46.756837 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.754129    2583 feature_gate.go:328] unrecognized feature gate: OSStreams
Aug 04 17:59:46.756837 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.754131    2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Aug 04 17:59:46.756837 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.754134    2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Aug 04 17:59:46.756837 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.754136    2583 feature_gate.go:328] unrecognized feature gate: Example
Aug 04 17:59:46.756837 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.754139    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Aug 04 17:59:46.756837 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.754141    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Aug 04 17:59:46.756837 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.754144    2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Aug 04 17:59:46.756837 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.754146    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Aug 04 17:59:46.756837 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.754149    2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Aug 04 17:59:46.756837 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.754152    2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Aug 04 17:59:46.756837 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.754154    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Aug 04 17:59:46.756837 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.754157    2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Aug 04 17:59:46.756837 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.754160    2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Aug 04 17:59:46.756837 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.754163    2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Aug 04 17:59:46.757332 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.754166    2583 feature_gate.go:328] unrecognized feature gate: NewOLM
Aug 04 17:59:46.757332 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.754170    2583 feature_gate.go:328] unrecognized feature gate: PinnedImages
Aug 04 17:59:46.757332 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.754172    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Aug 04 17:59:46.757332 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.754176    2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Aug 04 17:59:46.757332 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.754178    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Aug 04 17:59:46.757332 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.754181    2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Aug 04 17:59:46.757332 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.754183    2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Aug 04 17:59:46.757332 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.754186    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Aug 04 17:59:46.757332 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.754189    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Aug 04 17:59:46.757332 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.754191    2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Aug 04 17:59:46.757332 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.754194    2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Aug 04 17:59:46.757332 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.754196    2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Aug 04 17:59:46.757332 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.754199    2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Aug 04 17:59:46.757332 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.754202    2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Aug 04 17:59:46.757332 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.754204    2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Aug 04 17:59:46.757332 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.754207    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Aug 04 17:59:46.757332 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.754209    2583 feature_gate.go:328] unrecognized feature gate: SignatureStores
Aug 04 17:59:46.757332 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.754212    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Aug 04 17:59:46.757332 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.754214    2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Aug 04 17:59:46.757332 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.754217    2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Aug 04 17:59:46.757836 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.754220    2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Aug 04 17:59:46.757836 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.754636    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Aug 04 17:59:46.757836 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.754642    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Aug 04 17:59:46.757836 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.754645    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Aug 04 17:59:46.757836 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.754649    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Aug 04 17:59:46.757836 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.754652    2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Aug 04 17:59:46.757836 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.754655    2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Aug 04 17:59:46.757836 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.754658    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Aug 04 17:59:46.757836 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.754661    2583 feature_gate.go:328] unrecognized feature gate: DualReplica
Aug 04 17:59:46.757836 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.754664    2583 feature_gate.go:328] unrecognized feature gate: SignatureStores
Aug 04 17:59:46.757836 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.754667    2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Aug 04 17:59:46.757836 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.754669    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Aug 04 17:59:46.757836 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.754672    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Aug 04 17:59:46.757836 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.754676    2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Aug 04 17:59:46.757836 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.754680    2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Aug 04 17:59:46.757836 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.754684    2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Aug 04 17:59:46.757836 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.754687    2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Aug 04 17:59:46.757836 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.754689    2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Aug 04 17:59:46.757836 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.754692    2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Aug 04 17:59:46.758302 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.754695    2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Aug 04 17:59:46.758302 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.754697    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Aug 04 17:59:46.758302 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.754700    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Aug 04 17:59:46.758302 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.754702    2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Aug 04 17:59:46.758302 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.754705    2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Aug 04 17:59:46.758302 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.754708    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Aug 04 17:59:46.758302 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.754710    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Aug 04 17:59:46.758302 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.754713    2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Aug 04 17:59:46.758302 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.754716    2583 feature_gate.go:328] unrecognized feature gate: OSStreams
Aug 04 17:59:46.758302 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.754718    2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Aug 04 17:59:46.758302 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.754721    2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Aug 04 17:59:46.758302 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.754723    2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Aug 04 17:59:46.758302 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.754726    2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Aug 04 17:59:46.758302 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.754728    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Aug 04 17:59:46.758302 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.754731    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Aug 04 17:59:46.758302 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.754733    2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Aug 04 17:59:46.758302 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.754736    2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Aug 04 17:59:46.758302 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.754738    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Aug 04 17:59:46.758302 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.754741    2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Aug 04 17:59:46.758302 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.754743    2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Aug 04 17:59:46.758799 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.754746    2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Aug 04 17:59:46.758799 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.754749    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Aug 04 17:59:46.758799 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.754752    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Aug 04 17:59:46.758799 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.754754    2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Aug 04 17:59:46.758799 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.754757    2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Aug 04 17:59:46.758799 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.754762    2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Aug 04 17:59:46.758799 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.754765    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Aug 04 17:59:46.758799 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.754768    2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Aug 04 17:59:46.758799 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.754771    2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Aug 04 17:59:46.758799 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.754774    2583 feature_gate.go:328] unrecognized feature gate: PinnedImages
Aug 04 17:59:46.758799 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.754777    2583 feature_gate.go:328] unrecognized feature gate: EventTTL
Aug 04 17:59:46.758799 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.754780    2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Aug 04 17:59:46.758799 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.754782    2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Aug 04 17:59:46.758799 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.754785    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Aug 04 17:59:46.758799 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.754788    2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Aug 04 17:59:46.758799 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.754790    2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Aug 04 17:59:46.758799 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.754793    2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Aug 04 17:59:46.758799 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.754796    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Aug 04 17:59:46.758799 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.754798    2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Aug 04 17:59:46.759290 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.754801    2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Aug 04 17:59:46.759290 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.754803    2583 feature_gate.go:328] unrecognized feature gate: OVNObservability
Aug 04 17:59:46.759290 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.754806    2583 feature_gate.go:328] unrecognized feature gate: Example
Aug 04 17:59:46.759290 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.754808    2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Aug 04 17:59:46.759290 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.754811    2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Aug 04 17:59:46.759290 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.754814    2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Aug 04 17:59:46.759290 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.754816    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Aug 04 17:59:46.759290 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.754819    2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Aug 04 17:59:46.759290 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.754821    2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Aug 04 17:59:46.759290 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.754824    2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Aug 04 17:59:46.759290 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.754826    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Aug 04 17:59:46.759290 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.754829    2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Aug 04 17:59:46.759290 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.754833    2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Aug 04 17:59:46.759290 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.754836    2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Aug 04 17:59:46.759290 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.754839    2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Aug 04 17:59:46.759290 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.754841    2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Aug 04 17:59:46.759290 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.754844    2583 feature_gate.go:328] unrecognized feature gate: NewOLM
Aug 04 17:59:46.759290 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.754846    2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Aug 04 17:59:46.759290 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.754850    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Aug 04 17:59:46.759290 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.754853    2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Aug 04 17:59:46.759789 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.754855    2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Aug 04 17:59:46.759789 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.754858    2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Aug 04 17:59:46.759789 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.754861    2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Aug 04 17:59:46.759789 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.754863    2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Aug 04 17:59:46.759789 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.754866    2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Aug 04 17:59:46.759789 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.754869    2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Aug 04 17:59:46.759789 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.754871    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Aug 04 17:59:46.759789 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.754874    2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Aug 04 17:59:46.759789 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.754878    2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Aug 04 17:59:46.759789 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.754881    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Aug 04 17:59:46.759789 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.754884    2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Aug 04 17:59:46.759789 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.754886    2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Aug 04 17:59:46.759789 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.754889    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Aug 04 17:59:46.759789 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.754892    2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Aug 04 17:59:46.759789 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.754895    2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Aug 04 17:59:46.759789 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.754897    2583 feature_gate.go:328] unrecognized feature gate: Example2
Aug 04 17:59:46.759789 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.754900    2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Aug 04 17:59:46.759789 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.754904    2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Aug 04 17:59:46.759789 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.754908    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Aug 04 17:59:46.760263 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.754911    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Aug 04 17:59:46.760263 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.754914    2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Aug 04 17:59:46.760263 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.754916    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Aug 04 17:59:46.760263 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.754982    2583 flags.go:64] FLAG: --address="0.0.0.0"
Aug 04 17:59:46.760263 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.754988    2583 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Aug 04 17:59:46.760263 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.754996    2583 flags.go:64] FLAG: --anonymous-auth="true"
Aug 04 17:59:46.760263 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.755000    2583 flags.go:64] FLAG: --application-metrics-count-limit="100"
Aug 04 17:59:46.760263 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.755004    2583 flags.go:64] FLAG: --authentication-token-webhook="false"
Aug 04 17:59:46.760263 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.755008    2583 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Aug 04 17:59:46.760263 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.755012    2583 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Aug 04 17:59:46.760263 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.755016    2583 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Aug 04 17:59:46.760263 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.755020    2583 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Aug 04 17:59:46.760263 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.755023    2583 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Aug 04 17:59:46.760263 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.755027    2583 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Aug 04 17:59:46.760263 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.755030    2583 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Aug 04 17:59:46.760263 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.755034    2583 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Aug 04 17:59:46.760263 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.755038    2583 flags.go:64] FLAG: --cgroup-root=""
Aug 04 17:59:46.760263 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.755041    2583 flags.go:64] FLAG: --cgroups-per-qos="true"
Aug 04 17:59:46.760263 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.755044    2583 flags.go:64] FLAG: --client-ca-file=""
Aug 04 17:59:46.760263 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.755047    2583 flags.go:64] FLAG: --cloud-provider="external"
Aug 04 17:59:46.760263 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.755050    2583 flags.go:64] FLAG: --cluster-dns="[]"
Aug 04 17:59:46.760263 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.755054    2583 flags.go:64] FLAG: --cluster-domain=""
Aug 04 17:59:46.760263 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.755057    2583 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Aug 04 17:59:46.760263 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.755060    2583 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Aug 04 17:59:46.760946 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.755063    2583 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Aug 04 17:59:46.760946 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.755067    2583 flags.go:64] FLAG: --container-log-max-files="5"
Aug 04 17:59:46.760946 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.755071    2583 flags.go:64] FLAG: --container-log-max-size="10Mi"
Aug 04 17:59:46.760946 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.755074    2583 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Aug 04 17:59:46.760946 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.755077    2583 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Aug 04 17:59:46.760946 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.755080    2583 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Aug 04 17:59:46.760946 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.755083    2583 flags.go:64] FLAG: --contention-profiling="false"
Aug 04 17:59:46.760946 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.755086    2583 flags.go:64] FLAG: --cpu-cfs-quota="true"
Aug 04 17:59:46.760946 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.755090    2583 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Aug 04 17:59:46.760946 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.755093    2583 flags.go:64] FLAG: --cpu-manager-policy="none"
Aug 04 17:59:46.760946 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.755096    2583 flags.go:64] FLAG: --cpu-manager-policy-options=""
Aug 04 17:59:46.760946 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.755101    2583 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Aug 04 17:59:46.760946 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.755104    2583 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Aug 04 17:59:46.760946 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.755107    2583 flags.go:64] FLAG: --enable-debugging-handlers="true"
Aug 04 17:59:46.760946 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.755110    2583 flags.go:64] FLAG: --enable-load-reader="false"
Aug 04 17:59:46.760946 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.755113    2583 flags.go:64] FLAG: --enable-server="true"
Aug 04 17:59:46.760946 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.755116    2583 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Aug 04 17:59:46.760946 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.755120    2583 flags.go:64] FLAG: --event-burst="100"
Aug 04 17:59:46.760946 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.755124    2583 flags.go:64] FLAG: --event-qps="50"
Aug 04 17:59:46.760946 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.755128    2583 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Aug 04 17:59:46.760946 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.755131    2583 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Aug 04 17:59:46.760946 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.755134    2583 flags.go:64] FLAG: --eviction-hard=""
Aug 04 17:59:46.760946 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.755138    2583 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Aug 04 17:59:46.760946 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.755141    2583 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Aug 04 17:59:46.761543 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.755144    2583 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Aug 04 17:59:46.761543 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.755147    2583 flags.go:64] FLAG: --eviction-soft=""
Aug 04 17:59:46.761543 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.755151    2583 flags.go:64] FLAG: --eviction-soft-grace-period=""
Aug 04 17:59:46.761543 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.755153    2583 flags.go:64] FLAG: --exit-on-lock-contention="false"
Aug 04 17:59:46.761543 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.755156    2583 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Aug 04 17:59:46.761543 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.755159    2583 flags.go:64] FLAG: --experimental-mounter-path=""
Aug 04 17:59:46.761543 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.755162    2583 flags.go:64] FLAG: --fail-cgroupv1="false"
Aug 04 17:59:46.761543 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.755165    2583 flags.go:64] FLAG: --fail-swap-on="true"
Aug 04 17:59:46.761543 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.755168    2583 flags.go:64] FLAG: --feature-gates=""
Aug 04 17:59:46.761543 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.755172    2583 flags.go:64] FLAG: --file-check-frequency="20s"
Aug 04 17:59:46.761543 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.755174    2583 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Aug 04 17:59:46.761543 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.755178    2583 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Aug 04 17:59:46.761543 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.755181    2583 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Aug 04 17:59:46.761543 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.755184    2583 flags.go:64] FLAG: --healthz-port="10248"
Aug 04 17:59:46.761543 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.755187    2583 flags.go:64] FLAG: --help="false"
Aug 04 17:59:46.761543 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.755189    2583 flags.go:64] FLAG: --hostname-override="ip-10-0-135-124.ec2.internal"
Aug 04 17:59:46.761543 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.755192    2583 flags.go:64] FLAG: --housekeeping-interval="10s"
Aug 04 17:59:46.761543 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.755195    2583 flags.go:64] FLAG: --http-check-frequency="20s"
Aug 04 17:59:46.761543 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.755198    2583 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Aug 04 17:59:46.761543 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.755201    2583 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Aug 04 17:59:46.761543 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.755205    2583 flags.go:64] FLAG: --image-gc-high-threshold="85"
Aug 04 17:59:46.761543 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.755209    2583 flags.go:64] FLAG: --image-gc-low-threshold="80"
Aug 04 17:59:46.761543 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.755212    2583 flags.go:64] FLAG: --image-service-endpoint=""
Aug 04 17:59:46.762150 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.755214    2583 flags.go:64] FLAG: --kernel-memcg-notification="false"
Aug 04 17:59:46.762150 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.755217    2583 flags.go:64] FLAG: --kube-api-burst="100"
Aug 04 17:59:46.762150 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.755220    2583 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Aug 04 17:59:46.762150 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.755223    2583 flags.go:64] FLAG: --kube-api-qps="50"
Aug 04 17:59:46.762150 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.755227    2583 flags.go:64] FLAG: --kube-reserved=""
Aug 04 17:59:46.762150 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.755230    2583 flags.go:64] FLAG: --kube-reserved-cgroup=""
Aug 04 17:59:46.762150 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.755233    2583 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Aug 04 17:59:46.762150 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.755236    2583 flags.go:64] FLAG: --kubelet-cgroups=""
Aug 04 17:59:46.762150 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.755239    2583 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Aug 04 17:59:46.762150 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.755242    2583 flags.go:64] FLAG: --lock-file=""
Aug 04 17:59:46.762150 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.755251    2583 flags.go:64] FLAG: --log-cadvisor-usage="false"
Aug 04 17:59:46.762150 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.755254    2583 flags.go:64] FLAG: --log-flush-frequency="5s"
Aug 04 17:59:46.762150 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.755258    2583 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Aug 04 17:59:46.762150 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.755263    2583 flags.go:64] FLAG: --log-json-split-stream="false"
Aug 04 17:59:46.762150 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.755266    2583 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Aug 04 17:59:46.762150 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.755268    2583 flags.go:64] FLAG: --log-text-split-stream="false"
Aug 04 17:59:46.762150 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.755271    2583 flags.go:64] FLAG: --logging-format="text"
Aug 04 17:59:46.762150 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.755274    2583 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Aug 04 17:59:46.762150 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.755277    2583 flags.go:64] FLAG: --make-iptables-util-chains="true"
Aug 04 17:59:46.762150 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.755280    2583 flags.go:64] FLAG: --manifest-url=""
Aug 04 17:59:46.762150 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.755283    2583 flags.go:64] FLAG: --manifest-url-header=""
Aug 04 17:59:46.762150 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.755287    2583 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Aug 04 17:59:46.762150 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.755290    2583 flags.go:64] FLAG: --max-open-files="1000000"
Aug 04 17:59:46.762150 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.755294    2583 flags.go:64] FLAG: --max-pods="110"
Aug 04 17:59:46.762150 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.755297    2583 flags.go:64] FLAG: --maximum-dead-containers="-1"
Aug 04 17:59:46.762815 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.755300    2583 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Aug 04 17:59:46.762815 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.755303    2583 flags.go:64] FLAG: --memory-manager-policy="None"
Aug 04 17:59:46.762815 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.755305    2583 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Aug 04 17:59:46.762815 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.755308    2583 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Aug 04 17:59:46.762815 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.755311    2583 flags.go:64] FLAG: --node-ip="0.0.0.0"
Aug 04 17:59:46.762815 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.755315    2583 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Aug 04 17:59:46.762815 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.755322    2583 flags.go:64] FLAG: --node-status-max-images="50"
Aug 04 17:59:46.762815 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.755325    2583 flags.go:64] FLAG: --node-status-update-frequency="10s"
Aug 04 17:59:46.762815 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.755329    2583 flags.go:64] FLAG: --oom-score-adj="-999"
Aug 04 17:59:46.762815 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.755332    2583 flags.go:64] FLAG: --pod-cidr=""
Aug 04 17:59:46.762815 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.755334    2583 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:267c8c33005f5dc3d4121af550b1c396e30e2928a13624d7362adfbfb7f440f8"
Aug 04 17:59:46.762815 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.755340    2583 flags.go:64] FLAG: --pod-manifest-path=""
Aug 04 17:59:46.762815 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.755344    2583 flags.go:64] FLAG: --pod-max-pids="-1"
Aug 04 17:59:46.762815 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.755347    2583 flags.go:64] FLAG: --pods-per-core="0"
Aug 04 17:59:46.762815 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.755350    2583 flags.go:64] FLAG: --port="10250"
Aug 04 17:59:46.762815 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.755353    2583 flags.go:64] FLAG: --protect-kernel-defaults="false"
Aug 04 17:59:46.762815 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.755355    2583 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-00e2a16dbc62caffa"
Aug 04 17:59:46.762815 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.755358    2583 flags.go:64] FLAG: --qos-reserved=""
Aug 04 17:59:46.762815 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.755363    2583 flags.go:64] FLAG: --read-only-port="10255"
Aug 04 17:59:46.762815 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.755365    2583 flags.go:64] FLAG: --register-node="true"
Aug 04 17:59:46.762815 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.755368    2583 flags.go:64] FLAG: --register-with-taints=""
Aug 04 17:59:46.762815 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.755372    2583 flags.go:64] FLAG: --registry-burst="10"
Aug 04 17:59:46.762815 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.755375    2583 flags.go:64] FLAG: --registry-qps="5"
Aug 04 17:59:46.762815 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.755378    2583 flags.go:64] FLAG: --reserved-cpus=""
Aug 04 17:59:46.763404 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.755380    2583 flags.go:64] FLAG: --reserved-memory=""
Aug 04 17:59:46.763404 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.755384    2583 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Aug 04 17:59:46.763404 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.755388    2583 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Aug 04 17:59:46.763404 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.755391    2583 flags.go:64] FLAG: --rotate-certificates="false"
Aug 04 17:59:46.763404 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.755394    2583 flags.go:64] FLAG: --rotate-server-certificates="false"
Aug 04 17:59:46.763404 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.755397    2583 flags.go:64] FLAG: --runonce="false"
Aug 04 17:59:46.763404 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.755400    2583 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Aug 04 17:59:46.763404 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.755403    2583 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Aug 04 17:59:46.763404 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.755406    2583 flags.go:64] FLAG: --seccomp-default="false"
Aug 04 17:59:46.763404 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.755409    2583 flags.go:64] FLAG: --serialize-image-pulls="true"
Aug 04 17:59:46.763404 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.755412    2583 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Aug 04 17:59:46.763404 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.755415    2583 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Aug 04 17:59:46.763404 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.755418    2583 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Aug 04 17:59:46.763404 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.755421    2583 flags.go:64] FLAG: --storage-driver-password="root"
Aug 04 17:59:46.763404 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.755424    2583 flags.go:64] FLAG: --storage-driver-secure="false"
Aug 04 17:59:46.763404 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.755427    2583 flags.go:64] FLAG: --storage-driver-table="stats"
Aug 04 17:59:46.763404 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.755430    2583 flags.go:64] FLAG: --storage-driver-user="root"
Aug 04 17:59:46.763404 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.755433    2583 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Aug 04 17:59:46.763404 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.755436    2583 flags.go:64] FLAG: --sync-frequency="1m0s"
Aug 04 17:59:46.763404 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.755439    2583 flags.go:64] FLAG: --system-cgroups=""
Aug 04 17:59:46.763404 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.755443    2583 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Aug 04 17:59:46.763404 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.755448    2583 flags.go:64] FLAG: --system-reserved-cgroup=""
Aug 04 17:59:46.763404 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.755451    2583 flags.go:64] FLAG: --tls-cert-file=""
Aug 04 17:59:46.763404 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.755453    2583 flags.go:64] FLAG: --tls-cipher-suites="[]"
Aug 04 17:59:46.763404 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.755458    2583 flags.go:64] FLAG: --tls-min-version=""
Aug 04 17:59:46.764030 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.755461    2583 flags.go:64] FLAG: --tls-private-key-file=""
Aug 04 17:59:46.764030 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.755464    2583 flags.go:64] FLAG: --topology-manager-policy="none"
Aug 04 17:59:46.764030 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.755480    2583 flags.go:64] FLAG: --topology-manager-policy-options=""
Aug 04 17:59:46.764030 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.755483    2583 flags.go:64] FLAG: --topology-manager-scope="container"
Aug 04 17:59:46.764030 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.755487    2583 flags.go:64] FLAG: --v="2"
Aug 04 17:59:46.764030 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.755491    2583 flags.go:64] FLAG: --version="false"
Aug 04 17:59:46.764030 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.755498    2583 flags.go:64] FLAG: --vmodule=""
Aug 04 17:59:46.764030 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.755502    2583 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Aug 04 17:59:46.764030 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.755505    2583 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Aug 04 17:59:46.764030 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.755645    2583 feature_gate.go:328] unrecognized feature gate: OSStreams
Aug 04 17:59:46.764030 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.755649    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Aug 04 17:59:46.764030 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.755652    2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Aug 04 17:59:46.764030 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.755655    2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Aug 04 17:59:46.764030 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.755658    2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Aug 04 17:59:46.764030 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.755661    2583 feature_gate.go:328] unrecognized feature gate: OVNObservability
Aug 04 17:59:46.764030 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.755663    2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Aug 04 17:59:46.764030 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.755666    2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Aug 04 17:59:46.764030 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.755669    2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Aug 04 17:59:46.764030 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.755671    2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Aug 04 17:59:46.764030 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.755674    2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Aug 04 17:59:46.764030 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.755677    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Aug 04 17:59:46.764030 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.755680    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Aug 04 17:59:46.764645 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.755683    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Aug 04 17:59:46.764645 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.755685    2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Aug 04 17:59:46.764645 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.755688    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Aug 04 17:59:46.764645 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.755692    2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Aug 04 17:59:46.764645 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.755695    2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Aug 04 17:59:46.764645 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.755699    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Aug 04 17:59:46.764645 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.755702    2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Aug 04 17:59:46.764645 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.755705    2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Aug 04 17:59:46.764645 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.755708    2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Aug 04 17:59:46.764645 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.755710    2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Aug 04 17:59:46.764645 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.755713    2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Aug 04 17:59:46.764645 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.755717    2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Aug 04 17:59:46.764645 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.755721    2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Aug 04 17:59:46.764645 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.755730    2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Aug 04 17:59:46.764645 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.755734    2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Aug 04 17:59:46.764645 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.755737    2583 feature_gate.go:328] unrecognized feature gate: Example2
Aug 04 17:59:46.764645 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.755739    2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Aug 04 17:59:46.764645 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.755742    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Aug 04 17:59:46.764645 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.755744    2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Aug 04 17:59:46.765125 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.755747    2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Aug 04 17:59:46.765125 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.755749    2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Aug 04 17:59:46.765125 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.755752    2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Aug 04 17:59:46.765125 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.755754    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Aug 04 17:59:46.765125 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.755757    2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Aug 04 17:59:46.765125 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.755759    2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Aug 04 17:59:46.765125 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.755762    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Aug 04 17:59:46.765125 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.755765    2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Aug 04 17:59:46.765125 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.755768    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Aug 04 17:59:46.765125 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.755770    2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Aug 04 17:59:46.765125 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.755773    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Aug 04 17:59:46.765125 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.755776    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Aug 04 17:59:46.765125 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.755779    2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Aug 04 17:59:46.765125 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.755781    2583 feature_gate.go:328] unrecognized feature gate: NewOLM
Aug 04 17:59:46.765125 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.755784    2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Aug 04 17:59:46.765125 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.755786    2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Aug 04 17:59:46.765125 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.755789    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Aug 04 17:59:46.765125 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.755791    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Aug 04 17:59:46.765125 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.755795    2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Aug 04 17:59:46.765125 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.755797    2583 feature_gate.go:328] unrecognized feature gate: EventTTL
Aug 04 17:59:46.765623 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.755800    2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Aug 04 17:59:46.765623 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.755803    2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Aug 04 17:59:46.765623 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.755805    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Aug 04 17:59:46.765623 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.755808    2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Aug 04 17:59:46.765623 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.755811    2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Aug 04 17:59:46.765623 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.755813    2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Aug 04 17:59:46.765623 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.755816    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Aug 04 17:59:46.765623 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.755819    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Aug 04 17:59:46.765623 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.755822    2583 feature_gate.go:328] unrecognized feature gate: SignatureStores
Aug 04 17:59:46.765623 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.755824    2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Aug 04 17:59:46.765623 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.755827    2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Aug 04 17:59:46.765623 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.755830    2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Aug 04 17:59:46.765623 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.755832    2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Aug 04 17:59:46.765623 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.755835    2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Aug 04 17:59:46.765623 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.755837    2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Aug 04 17:59:46.765623 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.755840    2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Aug 04 17:59:46.765623 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.755843    2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Aug 04 17:59:46.765623 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.755845    2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Aug 04 17:59:46.765623 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.755848    2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Aug 04 17:59:46.765623 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.755851    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Aug 04 17:59:46.766103 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.755853    2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Aug 04 17:59:46.766103 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.755856    2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Aug 04 17:59:46.766103 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.755858    2583 feature_gate.go:328] unrecognized feature gate: PinnedImages
Aug 04 17:59:46.766103 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.755861    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Aug 04 17:59:46.766103 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.755864    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Aug 04 17:59:46.766103 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.755866    2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Aug 04 17:59:46.766103 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.755869    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Aug 04 17:59:46.766103 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.755871    2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Aug 04 17:59:46.766103 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.755874    2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Aug 04 17:59:46.766103 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.755876    2583 feature_gate.go:328] unrecognized feature gate: DualReplica
Aug 04 17:59:46.766103 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.755881    2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Aug 04 17:59:46.766103 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.755884    2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Aug 04 17:59:46.766103 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.755887    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Aug 04 17:59:46.766103 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.755889    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Aug 04 17:59:46.766103 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.755892    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Aug 04 17:59:46.766103 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.755896    2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Aug 04 17:59:46.766103 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.755900    2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Aug 04 17:59:46.766103 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.755903    2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Aug 04 17:59:46.766103 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.755906    2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Aug 04 17:59:46.766575 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.755908    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Aug 04 17:59:46.766575 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.755911    2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Aug 04 17:59:46.766575 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.755914    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Aug 04 17:59:46.766575 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.755917    2583 feature_gate.go:328] unrecognized feature gate: Example
Aug 04 17:59:46.766575 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.755919    2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Aug 04 17:59:46.766575 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.755922    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Aug 04 17:59:46.766575 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.755924    2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Aug 04 17:59:46.766575 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.755928    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Aug 04 17:59:46.766575 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.756648    2583 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Aug 04 17:59:46.766575 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.763456    2583 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Aug 04 17:59:46.766575 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.763488    2583 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Aug 04 17:59:46.766575 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.764304    2583 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Aug 04 17:59:46.766575 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.764312    2583 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Aug 04 17:59:46.766575 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.764380    2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Aug 04 17:59:46.766921 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.764385    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Aug 04 17:59:46.766921 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.764390    2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Aug 04 17:59:46.766921 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.764393    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Aug 04 17:59:46.766921 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.764396    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Aug 04 17:59:46.766921 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.764399    2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Aug 04 17:59:46.766921 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.764402    2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Aug 04 17:59:46.766921 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.764405    2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Aug 04 17:59:46.766921 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.764408    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Aug 04 17:59:46.766921 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.764411    2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Aug 04 17:59:46.766921 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.764414    2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Aug 04 17:59:46.766921 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.764417    2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Aug 04 17:59:46.766921 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.764419    2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Aug 04 17:59:46.766921 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.764423    2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Aug 04 17:59:46.766921 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.764425    2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Aug 04 17:59:46.766921 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.764428    2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Aug 04 17:59:46.766921 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.764431    2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Aug 04 17:59:46.766921 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.764433    2583 feature_gate.go:328] unrecognized feature gate: Example2
Aug 04 17:59:46.766921 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.764436    2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Aug 04 17:59:46.766921 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.764439    2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Aug 04 17:59:46.766921 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.764441    2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Aug 04 17:59:46.767396 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.764444    2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Aug 04 17:59:46.767396 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.764447    2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Aug 04 17:59:46.767396 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.764449    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Aug 04 17:59:46.767396 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.764452    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Aug 04 17:59:46.767396 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.764454    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Aug 04 17:59:46.767396 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.764457    2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Aug 04 17:59:46.767396 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.764460    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Aug 04 17:59:46.767396 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.764463    2583 feature_gate.go:328] unrecognized feature gate: PinnedImages
Aug 04 17:59:46.767396 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.764481    2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Aug 04 17:59:46.767396 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.764484    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Aug 04 17:59:46.767396 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.764487    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Aug 04 17:59:46.767396 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.764489    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Aug 04 17:59:46.767396 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.764492    2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Aug 04 17:59:46.767396 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.764495    2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Aug 04 17:59:46.767396 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.764498    2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Aug 04 17:59:46.767396 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.764500    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Aug 04 17:59:46.767396 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.764503    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Aug 04 17:59:46.767396 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.764506    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Aug 04 17:59:46.767396 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.764508    2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Aug 04 17:59:46.767396 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.764511    2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Aug 04 17:59:46.767917 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.764514    2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Aug 04 17:59:46.767917 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.764516    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Aug 04 17:59:46.767917 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.764520    2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Aug 04 17:59:46.767917 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.764524    2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Aug 04 17:59:46.767917 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.764526    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Aug 04 17:59:46.767917 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.764529    2583 feature_gate.go:328] unrecognized feature gate: Example
Aug 04 17:59:46.767917 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.764532    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Aug 04 17:59:46.767917 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.764535    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Aug 04 17:59:46.767917 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.764537    2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Aug 04 17:59:46.767917 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.764540    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Aug 04 17:59:46.767917 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.764542    2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Aug 04 17:59:46.767917 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.764545    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Aug 04 17:59:46.767917 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.764548    2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Aug 04 17:59:46.767917 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.764550    2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Aug 04 17:59:46.767917 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.764553    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Aug 04 17:59:46.767917 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.764555    2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Aug 04 17:59:46.767917 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.764558    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Aug 04 17:59:46.767917 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.764561    2583 feature_gate.go:328] unrecognized feature gate: OVNObservability
Aug 04 17:59:46.767917 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.764564    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Aug 04 17:59:46.768406 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.764567    2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Aug 04 17:59:46.768406 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.764570    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Aug 04 17:59:46.768406 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.764574    2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Aug 04 17:59:46.768406 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.764577    2583 feature_gate.go:328] unrecognized feature gate: SignatureStores
Aug 04 17:59:46.768406 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.764580    2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Aug 04 17:59:46.768406 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.764583    2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Aug 04 17:59:46.768406 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.764586    2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Aug 04 17:59:46.768406 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.764588    2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Aug 04 17:59:46.768406 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.764591    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Aug 04 17:59:46.768406 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.764593    2583 feature_gate.go:328] unrecognized feature gate: NewOLM
Aug 04 17:59:46.768406 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.764596    2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Aug 04 17:59:46.768406 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.764599    2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Aug 04 17:59:46.768406 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.764601    2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Aug 04 17:59:46.768406 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.764604    2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Aug 04 17:59:46.768406 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.764606    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Aug 04 17:59:46.768406 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.764609    2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Aug 04 17:59:46.768406 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.764613    2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Aug 04 17:59:46.768406 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.764616    2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Aug 04 17:59:46.768406 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.764619    2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Aug 04 17:59:46.768938 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.764621    2583 feature_gate.go:328] unrecognized feature gate: OSStreams
Aug 04 17:59:46.768938 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.764624    2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Aug 04 17:59:46.768938 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.764626    2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Aug 04 17:59:46.768938 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.764629    2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Aug 04 17:59:46.768938 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.764631    2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Aug 04 17:59:46.768938 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.764634    2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Aug 04 17:59:46.768938 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.764636    2583 feature_gate.go:328] unrecognized feature gate: DualReplica
Aug 04 17:59:46.768938 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.764639    2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Aug 04 17:59:46.768938 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.764642    2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Aug 04 17:59:46.768938 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.764644    2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Aug 04 17:59:46.768938 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.764648    2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Aug 04 17:59:46.768938 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.764650    2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Aug 04 17:59:46.768938 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.764653    2583 feature_gate.go:328] unrecognized feature gate: EventTTL
Aug 04 17:59:46.768938 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.764656    2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Aug 04 17:59:46.768938 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.764659    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Aug 04 17:59:46.768938 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.764662    2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Aug 04 17:59:46.768938 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.764665    2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Aug 04 17:59:46.768938 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.764668    2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Aug 04 17:59:46.768938 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.764670    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Aug 04 17:59:46.768938 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.764673    2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Aug 04 17:59:46.769727 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.764678    2583 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Aug 04 17:59:46.769727 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.764784    2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Aug 04 17:59:46.769727 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.764789    2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Aug 04 17:59:46.769727 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.764792    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Aug 04 17:59:46.769727 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.764794    2583 feature_gate.go:328] unrecognized feature gate: SignatureStores
Aug 04 17:59:46.769727 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.764798    2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Aug 04 17:59:46.769727 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.764800    2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Aug 04 17:59:46.769727 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.764803    2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Aug 04 17:59:46.769727 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.764805    2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Aug 04 17:59:46.769727 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.764808    2583 feature_gate.go:328] unrecognized feature gate: EventTTL
Aug 04 17:59:46.769727 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.764810    2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Aug 04 17:59:46.769727 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.764813    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Aug 04 17:59:46.769727 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.764816    2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Aug 04 17:59:46.769727 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.764819    2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Aug 04 17:59:46.769727 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.764821    2583 feature_gate.go:328] unrecognized feature gate: Example
Aug 04 17:59:46.770158 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.764824    2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Aug 04 17:59:46.770158 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.764826    2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Aug 04 17:59:46.770158 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.764829    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Aug 04 17:59:46.770158 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.764831    2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Aug 04 17:59:46.770158 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.764834    2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Aug 04 17:59:46.770158 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.764836    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Aug 04 17:59:46.770158 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.764839    2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Aug 04 17:59:46.770158 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.764842    2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Aug 04 17:59:46.770158 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.764844    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Aug 04 17:59:46.770158 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.764847    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Aug 04 17:59:46.770158 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.764850    2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Aug 04 17:59:46.770158 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.764852    2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Aug 04 17:59:46.770158 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.764855    2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Aug 04 17:59:46.770158 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.764857    2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Aug 04 17:59:46.770158 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.764861    2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Aug 04 17:59:46.770158 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.764863    2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Aug 04 17:59:46.770158 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.764866    2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Aug 04 17:59:46.770158 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.764870    2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Aug 04 17:59:46.770158 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.764874    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Aug 04 17:59:46.770631 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.764877    2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Aug 04 17:59:46.770631 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.764880    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Aug 04 17:59:46.770631 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.764883    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Aug 04 17:59:46.770631 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.764886    2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Aug 04 17:59:46.770631 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.764889    2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Aug 04 17:59:46.770631 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.764892    2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Aug 04 17:59:46.770631 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.764895    2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Aug 04 17:59:46.770631 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.764897    2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Aug 04 17:59:46.770631 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.764900    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Aug 04 17:59:46.770631 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.764902    2583 feature_gate.go:328] unrecognized feature gate: OSStreams
Aug 04 17:59:46.770631 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.764905    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Aug 04 17:59:46.770631 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.764907    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Aug 04 17:59:46.770631 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.764910    2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Aug 04 17:59:46.770631 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.764912    2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Aug 04 17:59:46.770631 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.764915    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Aug 04 17:59:46.770631 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.764917    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Aug 04 17:59:46.770631 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.764920    2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Aug 04 17:59:46.770631 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.764922    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Aug 04 17:59:46.770631 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.764925    2583 feature_gate.go:328] unrecognized feature gate: Example2
Aug 04 17:59:46.770631 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.764928    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Aug 04 17:59:46.771382 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.764931    2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Aug 04 17:59:46.771382 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.764933    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Aug 04 17:59:46.771382 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.764936    2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Aug 04 17:59:46.771382 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.764938    2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Aug 04 17:59:46.771382 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.764941    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Aug 04 17:59:46.771382 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.764945    2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Aug 04 17:59:46.771382 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.764949    2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Aug 04 17:59:46.771382 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.764952    2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Aug 04 17:59:46.771382 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.764956    2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Aug 04 17:59:46.771382 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.764959    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Aug 04 17:59:46.771382 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.764962    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Aug 04 17:59:46.771382 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.764964    2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Aug 04 17:59:46.771382 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.764967    2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Aug 04 17:59:46.771382 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.764969    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Aug 04 17:59:46.771382 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.764972    2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Aug 04 17:59:46.771382 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.764974    2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Aug 04 17:59:46.771382 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.764977    2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Aug 04 17:59:46.771382 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.764980    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Aug 04 17:59:46.772130 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.764982    2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Aug 04 17:59:46.772130 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.764985    2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Aug 04 17:59:46.772130 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.764987    2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Aug 04 17:59:46.772130 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.764990    2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Aug 04 17:59:46.772130 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.764992    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Aug 04 17:59:46.772130 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.764994    2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Aug 04 17:59:46.772130 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.764997    2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Aug 04 17:59:46.772130 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.764999    2583 feature_gate.go:328] unrecognized feature gate: NewOLM
Aug 04 17:59:46.772130 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.765002    2583 feature_gate.go:328] unrecognized feature gate: DualReplica
Aug 04 17:59:46.772130 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.765004    2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Aug 04 17:59:46.772130 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.765007    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Aug 04 17:59:46.772130 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.765009    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Aug 04 17:59:46.772130 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.765012    2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Aug 04 17:59:46.772130 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.765014    2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Aug 04 17:59:46.772130 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.765017    2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Aug 04 17:59:46.772130 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.765019    2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Aug 04 17:59:46.772130 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.765022    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Aug 04 17:59:46.772130 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.765024    2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Aug 04 17:59:46.772130 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.765027    2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Aug 04 17:59:46.772130 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.765029    2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Aug 04 17:59:46.772918 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.765032    2583 feature_gate.go:328] unrecognized feature gate: PinnedImages
Aug 04 17:59:46.772918 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.765034    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Aug 04 17:59:46.772918 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.765037    2583 feature_gate.go:328] unrecognized feature gate: OVNObservability
Aug 04 17:59:46.772918 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.765040    2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Aug 04 17:59:46.772918 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.765042    2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Aug 04 17:59:46.772918 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.765044    2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Aug 04 17:59:46.772918 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.765047    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Aug 04 17:59:46.772918 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:46.765050    2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Aug 04 17:59:46.772918 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.765054    2583 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Aug 04 17:59:46.772918 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.765161    2583 server.go:962] "Client rotation is on, will bootstrap in background"
Aug 04 17:59:46.772918 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.767320    2583 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Aug 04 17:59:46.772918 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.769037    2583 server.go:1019] "Starting client certificate rotation"
Aug 04 17:59:46.772918 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.769697    2583 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Aug 04 17:59:46.772918 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.769753    2583 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Aug 04 17:59:46.773896 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.773879    2583 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Aug 04 17:59:46.776910 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.776894    2583 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Aug 04 17:59:46.783903 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.783884    2583 log.go:25] "Validated CRI v1 runtime API"
Aug 04 17:59:46.787045 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.787031    2583 log.go:25] "Validated CRI v1 image API"
Aug 04 17:59:46.787907 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.787891    2583 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Aug 04 17:59:46.791275 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.791256    2583 fs.go:135] Filesystem UUIDs: map[0722a960-ac21-450d-9b05-2c96cd36ef2f:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 b3d57fa4-a122-4075-a788-72c9f9f2b18a:/dev/nvme0n1p4]
Aug 04 17:59:46.791331 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.791275    2583 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Aug 04 17:59:46.795776 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.795670    2583 manager.go:216] Machine: {Timestamp:2026-08-04 17:59:46.795197996 +0000 UTC m=+0.376416565 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3116256 MemoryCapacity:33164492800 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec201c57bbfa8336722feb74bd652a0e SystemUUID:ec201c57-bbfa-8336-722f-eb74bd652a0e BootID:89d9de8e-b35b-46a6-a8cf-65a9fc95ab6f Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:fd:be:17:6a:69 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:fd:be:17:6a:69 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:d6:a2:66:a8:ba:90 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164492800 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Aug 04 17:59:46.795776 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.795769    2583 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Aug 04 17:59:46.795890 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.795834    2583 manager.go:232] Version: {KernelVersion:5.14.0-570.131.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260727-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Aug 04 17:59:46.800028 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.799996    2583 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Aug 04 17:59:46.800155 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.800029    2583 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-135-124.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Aug 04 17:59:46.801812 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.801801    2583 topology_manager.go:138] "Creating topology manager with none policy"
Aug 04 17:59:46.801847 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.801815    2583 container_manager_linux.go:309] "Creating device plugin manager"
Aug 04 17:59:46.801847 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.801822    2583 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Aug 04 17:59:46.802328 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.802309    2583 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Aug 04 17:59:46.802558 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.802547    2583 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Aug 04 17:59:46.803537 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.803527    2583 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Aug 04 17:59:46.808085 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.808073    2583 state_mem.go:36] "Initialized new in-memory state store"
Aug 04 17:59:46.808194 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.808185    2583 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Aug 04 17:59:46.808228 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.808212    2583 kubelet.go:486] "Attempting to sync node with API server"
Aug 04 17:59:46.808228 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.808222    2583 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Aug 04 17:59:46.808278 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.808236    2583 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Aug 04 17:59:46.808278 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.808245    2583 kubelet.go:398] "Adding apiserver pod source"
Aug 04 17:59:46.808278 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.808253    2583 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Aug 04 17:59:46.808577 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.808565    2583 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Aug 04 17:59:46.808624 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.808584    2583 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Aug 04 17:59:46.813078 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.813035    2583 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Aug 04 17:59:46.815116 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.815101    2583 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Aug 04 17:59:46.815176 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.815131    2583 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Aug 04 17:59:46.816349 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.816337    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Aug 04 17:59:46.816419 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.816352    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Aug 04 17:59:46.816419 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.816367    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Aug 04 17:59:46.816419 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.816374    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Aug 04 17:59:46.816419 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.816382    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Aug 04 17:59:46.816419 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.816389    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Aug 04 17:59:46.816419 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.816396    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Aug 04 17:59:46.816419 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.816402    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Aug 04 17:59:46.816419 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.816409    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Aug 04 17:59:46.816419 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.816414    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Aug 04 17:59:46.816419 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.816423    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Aug 04 17:59:46.816853 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.816428    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Aug 04 17:59:46.816853 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.816450    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Aug 04 17:59:46.816853 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.816456    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Aug 04 17:59:46.818973 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.818956    2583 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-135-124.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Aug 04 17:59:46.819084 ip-10-0-135-124 kubenswrapper[2583]: E0804 17:59:46.819063    2583 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-135-124.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Aug 04 17:59:46.819122 ip-10-0-135-124 kubenswrapper[2583]: E0804 17:59:46.819087    2583 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Aug 04 17:59:46.819375 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.819365    2583 server.go:1268] "Started kubelet"
Aug 04 17:59:46.819447 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.819424    2583 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Aug 04 17:59:46.819539 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.819494    2583 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Aug 04 17:59:46.819587 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.819562    2583 server_v1.go:49] "podresources" method="list" useActivePods=true
Aug 04 17:59:46.820266 ip-10-0-135-124 systemd[1]: Started Kubernetes Kubelet.
Aug 04 17:59:46.820845 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.820825    2583 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Aug 04 17:59:46.821604 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.821589    2583 server.go:310] "Adding debug handlers to kubelet server"
Aug 04 17:59:46.826713 ip-10-0-135-124 kubenswrapper[2583]: E0804 17:59:46.825563    2583 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-124.ec2.internal.18c8ac0452ad5889  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-124.ec2.internal,UID:ip-10-0-135-124.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-135-124.ec2.internal,},FirstTimestamp:2026-08-04 17:59:46.819340425 +0000 UTC m=+0.400558994,LastTimestamp:2026-08-04 17:59:46.819340425 +0000 UTC m=+0.400558994,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-124.ec2.internal,}"
Aug 04 17:59:46.826981 ip-10-0-135-124 kubenswrapper[2583]: E0804 17:59:46.826958    2583 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Aug 04 17:59:46.827930 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.827909    2583 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Aug 04 17:59:46.827930 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.827920    2583 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Aug 04 17:59:46.828066 ip-10-0-135-124 kubenswrapper[2583]: E0804 17:59:46.827990    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-124.ec2.internal\" not found"
Aug 04 17:59:46.828711 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.828628    2583 volume_manager.go:311] "The desired_state_of_world populator starts"
Aug 04 17:59:46.828711 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.828654    2583 volume_manager.go:313] "Starting Kubelet Volume Manager"
Aug 04 17:59:46.828711 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.828671    2583 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Aug 04 17:59:46.828874 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.828794    2583 reconstruct.go:97] "Volume reconstruction finished"
Aug 04 17:59:46.828874 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.828803    2583 reconciler.go:29] "Reconciler: start to sync state"
Aug 04 17:59:46.828944 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.828914    2583 factory.go:55] Registering systemd factory
Aug 04 17:59:46.828944 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.828931    2583 factory.go:223] Registration of the systemd container factory successfully
Aug 04 17:59:46.829154 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.829142    2583 factory.go:153] Registering CRI-O factory
Aug 04 17:59:46.829154 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.829156    2583 factory.go:223] Registration of the crio container factory successfully
Aug 04 17:59:46.829279 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.829195    2583 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Aug 04 17:59:46.829279 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.829212    2583 factory.go:103] Registering Raw factory
Aug 04 17:59:46.829279 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.829222    2583 manager.go:1196] Started watching for new ooms in manager
Aug 04 17:59:46.829588 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.829576    2583 manager.go:349] Starting recovery of all containers
Aug 04 17:59:46.833124 ip-10-0-135-124 kubenswrapper[2583]: E0804 17:59:46.833087    2583 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Aug 04 17:59:46.833233 ip-10-0-135-124 kubenswrapper[2583]: E0804 17:59:46.833194    2583 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-135-124.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Aug 04 17:59:46.840239 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.840221    2583 manager.go:354] Recovery completed
Aug 04 17:59:46.846635 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.846623    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Aug 04 17:59:46.848530 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.848458    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-124.ec2.internal" event="NodeHasSufficientMemory"
Aug 04 17:59:46.848583 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.848544    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-124.ec2.internal" event="NodeHasNoDiskPressure"
Aug 04 17:59:46.848583 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.848553    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-124.ec2.internal" event="NodeHasSufficientPID"
Aug 04 17:59:46.849039 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.849024    2583 cpu_manager.go:222] "Starting CPU manager" policy="none"
Aug 04 17:59:46.849039 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.849035    2583 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Aug 04 17:59:46.849039 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.849039    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Aug 04 17:59:46.849225 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.849049    2583 state_mem.go:36] "Initialized new in-memory state store"
Aug 04 17:59:46.849691 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.849672    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-124.ec2.internal" event="NodeHasSufficientMemory"
Aug 04 17:59:46.849758 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.849700    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-124.ec2.internal" event="NodeHasNoDiskPressure"
Aug 04 17:59:46.849758 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.849709    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-124.ec2.internal" event="NodeHasSufficientPID"
Aug 04 17:59:46.850875 ip-10-0-135-124 kubenswrapper[2583]: E0804 17:59:46.850801    2583 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-124.ec2.internal.18c8ac04546ac05b  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-124.ec2.internal,UID:ip-10-0-135-124.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-135-124.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-135-124.ec2.internal,},FirstTimestamp:2026-08-04 17:59:46.848530523 +0000 UTC m=+0.429749092,LastTimestamp:2026-08-04 17:59:46.848530523 +0000 UTC m=+0.429749092,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-124.ec2.internal,}"
Aug 04 17:59:46.851167 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.851157    2583 policy_none.go:49] "None policy: Start"
Aug 04 17:59:46.851199 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.851172    2583 memory_manager.go:187] "Starting memorymanager" policy="None"
Aug 04 17:59:46.851199 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.851182    2583 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Aug 04 17:59:46.853202 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.853182    2583 policy_none.go:47] "Start"
Aug 04 17:59:46.854464 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.854444    2583 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-xrjk4"
Aug 04 17:59:46.860304 ip-10-0-135-124 kubenswrapper[2583]: E0804 17:59:46.860247    2583 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-124.ec2.internal.18c8ac04546b063d  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-124.ec2.internal,UID:ip-10-0-135-124.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-135-124.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-135-124.ec2.internal,},FirstTimestamp:2026-08-04 17:59:46.848548413 +0000 UTC m=+0.429766982,LastTimestamp:2026-08-04 17:59:46.848548413 +0000 UTC m=+0.429766982,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-124.ec2.internal,}"
Aug 04 17:59:46.862258 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.862241    2583 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-xrjk4"
Aug 04 17:59:46.907195 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.896085    2583 manager.go:337] "Starting Device Plugin manager"
Aug 04 17:59:46.907195 ip-10-0-135-124 kubenswrapper[2583]: E0804 17:59:46.896235    2583 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Aug 04 17:59:46.907195 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.896248    2583 server.go:87] "Starting device plugin registration server"
Aug 04 17:59:46.907195 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.896435    2583 eviction_manager.go:189] "Eviction manager: starting control loop"
Aug 04 17:59:46.907195 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.896445    2583 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Aug 04 17:59:46.907195 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.896554    2583 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Aug 04 17:59:46.907195 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.896637    2583 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Aug 04 17:59:46.907195 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.896646    2583 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Aug 04 17:59:46.907195 ip-10-0-135-124 kubenswrapper[2583]: E0804 17:59:46.897513    2583 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Aug 04 17:59:46.907195 ip-10-0-135-124 kubenswrapper[2583]: E0804 17:59:46.897563    2583 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-135-124.ec2.internal\" not found"
Aug 04 17:59:46.934369 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.934344    2583 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Aug 04 17:59:46.935422 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.935399    2583 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Aug 04 17:59:46.935513 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.935446    2583 status_manager.go:244] "Starting to sync pod status with apiserver"
Aug 04 17:59:46.935513 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.935487    2583 kubelet.go:2448] "Starting kubelet main sync loop"
Aug 04 17:59:46.935600 ip-10-0-135-124 kubenswrapper[2583]: E0804 17:59:46.935525    2583 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Aug 04 17:59:46.938093 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.938073    2583 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Aug 04 17:59:46.997430 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.997377    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Aug 04 17:59:46.998149 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.998134    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-124.ec2.internal" event="NodeHasSufficientMemory"
Aug 04 17:59:46.998231 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.998168    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-124.ec2.internal" event="NodeHasNoDiskPressure"
Aug 04 17:59:46.998231 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.998183    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-124.ec2.internal" event="NodeHasSufficientPID"
Aug 04 17:59:46.998231 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:46.998212    2583 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-135-124.ec2.internal"
Aug 04 17:59:47.007188 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:47.007173    2583 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-135-124.ec2.internal"
Aug 04 17:59:47.007257 ip-10-0-135-124 kubenswrapper[2583]: E0804 17:59:47.007193    2583 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-135-124.ec2.internal\": node \"ip-10-0-135-124.ec2.internal\" not found"
Aug 04 17:59:47.036141 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:47.036121    2583 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-124.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-135-124.ec2.internal"]
Aug 04 17:59:47.036219 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:47.036174    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Aug 04 17:59:47.036912 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:47.036899    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-124.ec2.internal" event="NodeHasSufficientMemory"
Aug 04 17:59:47.036983 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:47.036923    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-124.ec2.internal" event="NodeHasNoDiskPressure"
Aug 04 17:59:47.036983 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:47.036933    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-124.ec2.internal" event="NodeHasSufficientPID"
Aug 04 17:59:47.039193 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:47.039181    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Aug 04 17:59:47.039333 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:47.039320    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-124.ec2.internal"
Aug 04 17:59:47.039369 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:47.039346    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Aug 04 17:59:47.039857 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:47.039840    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-124.ec2.internal" event="NodeHasSufficientMemory"
Aug 04 17:59:47.039956 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:47.039870    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-124.ec2.internal" event="NodeHasNoDiskPressure"
Aug 04 17:59:47.039956 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:47.039842    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-124.ec2.internal" event="NodeHasSufficientMemory"
Aug 04 17:59:47.039956 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:47.039908    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-124.ec2.internal" event="NodeHasNoDiskPressure"
Aug 04 17:59:47.039956 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:47.039881    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-124.ec2.internal" event="NodeHasSufficientPID"
Aug 04 17:59:47.039956 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:47.039938    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-124.ec2.internal" event="NodeHasSufficientPID"
Aug 04 17:59:47.042240 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:47.042225    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-124.ec2.internal"
Aug 04 17:59:47.042321 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:47.042249    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Aug 04 17:59:47.043245 ip-10-0-135-124 kubenswrapper[2583]: E0804 17:59:47.043229    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-124.ec2.internal\" not found"
Aug 04 17:59:47.043427 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:47.043410    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-124.ec2.internal" event="NodeHasSufficientMemory"
Aug 04 17:59:47.043528 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:47.043440    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-124.ec2.internal" event="NodeHasNoDiskPressure"
Aug 04 17:59:47.043528 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:47.043453    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-124.ec2.internal" event="NodeHasSufficientPID"
Aug 04 17:59:47.057046 ip-10-0-135-124 kubenswrapper[2583]: E0804 17:59:47.057029    2583 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-124.ec2.internal\" not found" node="ip-10-0-135-124.ec2.internal"
Aug 04 17:59:47.061288 ip-10-0-135-124 kubenswrapper[2583]: E0804 17:59:47.061274    2583 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-124.ec2.internal\" not found" node="ip-10-0-135-124.ec2.internal"
Aug 04 17:59:47.130902 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:47.130880    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4e7e6ce51e0af982f917261a0b47d870-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-124.ec2.internal\" (UID: \"4e7e6ce51e0af982f917261a0b47d870\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-124.ec2.internal"
Aug 04 17:59:47.130989 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:47.130909    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4e7e6ce51e0af982f917261a0b47d870-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-124.ec2.internal\" (UID: \"4e7e6ce51e0af982f917261a0b47d870\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-124.ec2.internal"
Aug 04 17:59:47.130989 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:47.130935    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f727e1c61c596ab926797040f56320d1-config\") pod \"kube-apiserver-proxy-ip-10-0-135-124.ec2.internal\" (UID: \"f727e1c61c596ab926797040f56320d1\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-124.ec2.internal"
Aug 04 17:59:47.143576 ip-10-0-135-124 kubenswrapper[2583]: E0804 17:59:47.143558    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-124.ec2.internal\" not found"
Aug 04 17:59:47.231916 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:47.231895    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f727e1c61c596ab926797040f56320d1-config\") pod \"kube-apiserver-proxy-ip-10-0-135-124.ec2.internal\" (UID: \"f727e1c61c596ab926797040f56320d1\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-124.ec2.internal"
Aug 04 17:59:47.231997 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:47.231932    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4e7e6ce51e0af982f917261a0b47d870-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-124.ec2.internal\" (UID: \"4e7e6ce51e0af982f917261a0b47d870\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-124.ec2.internal"
Aug 04 17:59:47.231997 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:47.231978    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4e7e6ce51e0af982f917261a0b47d870-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-124.ec2.internal\" (UID: \"4e7e6ce51e0af982f917261a0b47d870\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-124.ec2.internal"
Aug 04 17:59:47.232064 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:47.231997    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f727e1c61c596ab926797040f56320d1-config\") pod \"kube-apiserver-proxy-ip-10-0-135-124.ec2.internal\" (UID: \"f727e1c61c596ab926797040f56320d1\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-124.ec2.internal"
Aug 04 17:59:47.232064 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:47.232008    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4e7e6ce51e0af982f917261a0b47d870-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-124.ec2.internal\" (UID: \"4e7e6ce51e0af982f917261a0b47d870\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-124.ec2.internal"
Aug 04 17:59:47.232064 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:47.232034    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4e7e6ce51e0af982f917261a0b47d870-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-124.ec2.internal\" (UID: \"4e7e6ce51e0af982f917261a0b47d870\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-124.ec2.internal"
Aug 04 17:59:47.243981 ip-10-0-135-124 kubenswrapper[2583]: E0804 17:59:47.243960    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-124.ec2.internal\" not found"
Aug 04 17:59:47.344784 ip-10-0-135-124 kubenswrapper[2583]: E0804 17:59:47.344727    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-124.ec2.internal\" not found"
Aug 04 17:59:47.359893 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:47.359877    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-124.ec2.internal"
Aug 04 17:59:47.363415 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:47.363399    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-124.ec2.internal"
Aug 04 17:59:47.445219 ip-10-0-135-124 kubenswrapper[2583]: E0804 17:59:47.445194    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-124.ec2.internal\" not found"
Aug 04 17:59:47.545761 ip-10-0-135-124 kubenswrapper[2583]: E0804 17:59:47.545738    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-124.ec2.internal\" not found"
Aug 04 17:59:47.646258 ip-10-0-135-124 kubenswrapper[2583]: E0804 17:59:47.646188    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-124.ec2.internal\" not found"
Aug 04 17:59:47.746579 ip-10-0-135-124 kubenswrapper[2583]: E0804 17:59:47.746547    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-124.ec2.internal\" not found"
Aug 04 17:59:47.768328 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:47.768304    2583 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Aug 04 17:59:47.768791 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:47.768471    2583 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Aug 04 17:59:47.803617 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:47.803594    2583 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Aug 04 17:59:47.846664 ip-10-0-135-124 kubenswrapper[2583]: E0804 17:59:47.846638    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-124.ec2.internal\" not found"
Aug 04 17:59:47.867839 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:47.867808    2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-08-03 17:54:47 +0000 UTC" deadline="2028-04-16 01:36:26.301792892 +0000 UTC"
Aug 04 17:59:47.867839 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:47.867838    2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14887h36m38.433958279s"
Aug 04 17:59:47.871835 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:47.871815    2583 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Aug 04 17:59:47.928662 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:47.928597    2583 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-124.ec2.internal"
Aug 04 17:59:47.944445 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:47.944419    2583 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Aug 04 17:59:47.945747 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:47.945731    2583 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-124.ec2.internal"
Aug 04 17:59:47.955707 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:47.955688    2583 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Aug 04 17:59:47.989804 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:47.989776    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e7e6ce51e0af982f917261a0b47d870.slice/crio-17a7cf982218c7816a2d6a94653437ad4a525d8b01ccf7d05a1bb6123a07383f WatchSource:0}: Error finding container 17a7cf982218c7816a2d6a94653437ad4a525d8b01ccf7d05a1bb6123a07383f: Status 404 returned error can't find the container with id 17a7cf982218c7816a2d6a94653437ad4a525d8b01ccf7d05a1bb6123a07383f
Aug 04 17:59:47.998134 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:47.998120    2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Aug 04 17:59:48.002745 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:48.002715    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf727e1c61c596ab926797040f56320d1.slice/crio-2c2009093336b04ede0f138e853ce2f03d6a96be292e1b1fb787d686ab0dbc94 WatchSource:0}: Error finding container 2c2009093336b04ede0f138e853ce2f03d6a96be292e1b1fb787d686ab0dbc94: Status 404 returned error can't find the container with id 2c2009093336b04ede0f138e853ce2f03d6a96be292e1b1fb787d686ab0dbc94
Aug 04 17:59:48.121476 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:48.121442    2583 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Aug 04 17:59:48.810348 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:48.810315    2583 apiserver.go:52] "Watching apiserver"
Aug 04 17:59:48.820336 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:48.820308    2583 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Aug 04 17:59:48.824172 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:48.823753    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/node-ca-nz5rh","openshift-multus/network-metrics-daemon-qw2tv","openshift-network-diagnostics/network-check-target-jxnpj","openshift-ovn-kubernetes/ovnkube-node-mzl4m","kube-system/global-pull-secret-syncer-lqxxl","kube-system/kube-apiserver-proxy-ip-10-0-135-124.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-124.ec2.internal","openshift-multus/multus-additional-cni-plugins-c6m8x","openshift-multus/multus-jwcng","openshift-network-operator/iptables-alerter-rtcrf","kube-system/konnectivity-agent-ckl6v","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-57zmj","openshift-cluster-node-tuning-operator/tuned-7bwz5","openshift-dns/node-resolver-ggrqd"]
Aug 04 17:59:48.826716 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:48.826672    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-rtcrf"
Aug 04 17:59:48.829885 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:48.829038    2583 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Aug 04 17:59:48.829885 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:48.829661    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-7bwz5"
Aug 04 17:59:48.830066 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:48.830029    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-f4srf\""
Aug 04 17:59:48.830469 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:48.830134    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Aug 04 17:59:48.830469 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:48.830247    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Aug 04 17:59:48.830469 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:48.830326    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Aug 04 17:59:48.832518 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:48.832282    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-npbs9\""
Aug 04 17:59:48.832518 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:48.832409    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Aug 04 17:59:48.832518 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:48.832424    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Aug 04 17:59:48.833538 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:48.833296    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jxnpj"
Aug 04 17:59:48.833538 ip-10-0-135-124 kubenswrapper[2583]: E0804 17:59:48.833368    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jxnpj" podUID="43697de6-c2bd-48c6-81eb-1f37fa5d3174"
Aug 04 17:59:48.838230 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:48.837950    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-mzl4m"
Aug 04 17:59:48.842152 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:48.840300    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kj22l\" (UniqueName: \"kubernetes.io/projected/a8dd7a27-922c-4dde-bead-839291637a79-kube-api-access-kj22l\") pod \"iptables-alerter-rtcrf\" (UID: \"a8dd7a27-922c-4dde-bead-839291637a79\") " pod="openshift-network-operator/iptables-alerter-rtcrf"
Aug 04 17:59:48.842152 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:48.840337    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/de498784-c90b-4e01-bab8-e2ba98d49466-var-lib-kubelet\") pod \"tuned-7bwz5\" (UID: \"de498784-c90b-4e01-bab8-e2ba98d49466\") " pod="openshift-cluster-node-tuning-operator/tuned-7bwz5"
Aug 04 17:59:48.842152 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:48.840364    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/de498784-c90b-4e01-bab8-e2ba98d49466-etc-tuned\") pod \"tuned-7bwz5\" (UID: \"de498784-c90b-4e01-bab8-e2ba98d49466\") " pod="openshift-cluster-node-tuning-operator/tuned-7bwz5"
Aug 04 17:59:48.842152 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:48.840391    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87rpt\" (UniqueName: \"kubernetes.io/projected/43697de6-c2bd-48c6-81eb-1f37fa5d3174-kube-api-access-87rpt\") pod \"network-check-target-jxnpj\" (UID: \"43697de6-c2bd-48c6-81eb-1f37fa5d3174\") " pod="openshift-network-diagnostics/network-check-target-jxnpj"
Aug 04 17:59:48.842152 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:48.840433    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/de498784-c90b-4e01-bab8-e2ba98d49466-etc-sysconfig\") pod \"tuned-7bwz5\" (UID: \"de498784-c90b-4e01-bab8-e2ba98d49466\") " pod="openshift-cluster-node-tuning-operator/tuned-7bwz5"
Aug 04 17:59:48.842152 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:48.840456    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/de498784-c90b-4e01-bab8-e2ba98d49466-sys\") pod \"tuned-7bwz5\" (UID: \"de498784-c90b-4e01-bab8-e2ba98d49466\") " pod="openshift-cluster-node-tuning-operator/tuned-7bwz5"
Aug 04 17:59:48.842152 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:48.840481    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/de498784-c90b-4e01-bab8-e2ba98d49466-tmp\") pod \"tuned-7bwz5\" (UID: \"de498784-c90b-4e01-bab8-e2ba98d49466\") " pod="openshift-cluster-node-tuning-operator/tuned-7bwz5"
Aug 04 17:59:48.842152 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:48.840523    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a8dd7a27-922c-4dde-bead-839291637a79-host-slash\") pod \"iptables-alerter-rtcrf\" (UID: \"a8dd7a27-922c-4dde-bead-839291637a79\") " pod="openshift-network-operator/iptables-alerter-rtcrf"
Aug 04 17:59:48.842152 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:48.840550    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/de498784-c90b-4e01-bab8-e2ba98d49466-etc-systemd\") pod \"tuned-7bwz5\" (UID: \"de498784-c90b-4e01-bab8-e2ba98d49466\") " pod="openshift-cluster-node-tuning-operator/tuned-7bwz5"
Aug 04 17:59:48.842152 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:48.840572    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/de498784-c90b-4e01-bab8-e2ba98d49466-lib-modules\") pod \"tuned-7bwz5\" (UID: \"de498784-c90b-4e01-bab8-e2ba98d49466\") " pod="openshift-cluster-node-tuning-operator/tuned-7bwz5"
Aug 04 17:59:48.842152 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:48.840595    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/de498784-c90b-4e01-bab8-e2ba98d49466-host\") pod \"tuned-7bwz5\" (UID: \"de498784-c90b-4e01-bab8-e2ba98d49466\") " pod="openshift-cluster-node-tuning-operator/tuned-7bwz5"
Aug 04 17:59:48.842152 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:48.840631    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4v564\" (UniqueName: \"kubernetes.io/projected/de498784-c90b-4e01-bab8-e2ba98d49466-kube-api-access-4v564\") pod \"tuned-7bwz5\" (UID: \"de498784-c90b-4e01-bab8-e2ba98d49466\") " pod="openshift-cluster-node-tuning-operator/tuned-7bwz5"
Aug 04 17:59:48.842152 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:48.840683    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a8dd7a27-922c-4dde-bead-839291637a79-iptables-alerter-script\") pod \"iptables-alerter-rtcrf\" (UID: \"a8dd7a27-922c-4dde-bead-839291637a79\") " pod="openshift-network-operator/iptables-alerter-rtcrf"
Aug 04 17:59:48.842152 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:48.840762    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/de498784-c90b-4e01-bab8-e2ba98d49466-etc-modprobe-d\") pod \"tuned-7bwz5\" (UID: \"de498784-c90b-4e01-bab8-e2ba98d49466\") " pod="openshift-cluster-node-tuning-operator/tuned-7bwz5"
Aug 04 17:59:48.842152 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:48.840790    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/de498784-c90b-4e01-bab8-e2ba98d49466-etc-kubernetes\") pod \"tuned-7bwz5\" (UID: \"de498784-c90b-4e01-bab8-e2ba98d49466\") " pod="openshift-cluster-node-tuning-operator/tuned-7bwz5"
Aug 04 17:59:48.842152 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:48.840817    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/de498784-c90b-4e01-bab8-e2ba98d49466-etc-sysctl-d\") pod \"tuned-7bwz5\" (UID: \"de498784-c90b-4e01-bab8-e2ba98d49466\") " pod="openshift-cluster-node-tuning-operator/tuned-7bwz5"
Aug 04 17:59:48.842152 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:48.840835    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-9kpct\""
Aug 04 17:59:48.843756 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:48.840841    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/de498784-c90b-4e01-bab8-e2ba98d49466-etc-sysctl-conf\") pod \"tuned-7bwz5\" (UID: \"de498784-c90b-4e01-bab8-e2ba98d49466\") " pod="openshift-cluster-node-tuning-operator/tuned-7bwz5"
Aug 04 17:59:48.843756 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:48.840866    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/de498784-c90b-4e01-bab8-e2ba98d49466-run\") pod \"tuned-7bwz5\" (UID: \"de498784-c90b-4e01-bab8-e2ba98d49466\") " pod="openshift-cluster-node-tuning-operator/tuned-7bwz5"
Aug 04 17:59:48.843756 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:48.841050    2583 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Aug 04 17:59:48.843756 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:48.841263    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Aug 04 17:59:48.843756 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:48.841280    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Aug 04 17:59:48.843756 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:48.841645    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Aug 04 17:59:48.843756 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:48.841709    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Aug 04 17:59:48.843756 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:48.841896    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lqxxl"
Aug 04 17:59:48.843756 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:48.841905    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Aug 04 17:59:48.843756 ip-10-0-135-124 kubenswrapper[2583]: E0804 17:59:48.841957    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lqxxl" podUID="31fd88d9-8ce3-4056-a185-4427ec8939c1"
Aug 04 17:59:48.843756 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:48.842043    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Aug 04 17:59:48.843756 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:48.842611    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-ggrqd"
Aug 04 17:59:48.845318 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:48.845063    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Aug 04 17:59:48.845418 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:48.845346    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Aug 04 17:59:48.845418 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:48.845377    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-9rmkq\""
Aug 04 17:59:48.845762 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:48.845740    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-jwcng"
Aug 04 17:59:48.848271 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:48.848252    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-c6m8x"
Aug 04 17:59:48.848891 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:48.848786    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Aug 04 17:59:48.849009 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:48.848988    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Aug 04 17:59:48.849110 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:48.849093    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Aug 04 17:59:48.849820 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:48.848862    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-jc6bm\""
Aug 04 17:59:48.850077 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:48.850057    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Aug 04 17:59:48.853181 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:48.851748    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qw2tv"
Aug 04 17:59:48.853181 ip-10-0-135-124 kubenswrapper[2583]: E0804 17:59:48.851831    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qw2tv" podUID="0480eb75-3ca1-46ba-9606-ec9e98dc9c4c"
Aug 04 17:59:48.853181 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:48.851978    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Aug 04 17:59:48.853181 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:48.852237    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Aug 04 17:59:48.853181 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:48.852321    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-6lmdn\""
Aug 04 17:59:48.854634 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:48.854563    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-nz5rh"
Aug 04 17:59:48.856984 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:48.856963    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-ckl6v"
Aug 04 17:59:48.859675 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:48.859653    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-57zmj"
Aug 04 17:59:48.862613 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:48.862592    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Aug 04 17:59:48.862710 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:48.862618    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-pjmtd\""
Aug 04 17:59:48.862771 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:48.862747    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Aug 04 17:59:48.862837 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:48.862820    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Aug 04 17:59:48.863043 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:48.863023    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Aug 04 17:59:48.863148 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:48.863098    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Aug 04 17:59:48.863227 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:48.863211    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-n48gm\""
Aug 04 17:59:48.863588 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:48.863566    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Aug 04 17:59:48.863796 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:48.863742    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-nccl5\""
Aug 04 17:59:48.863864 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:48.863845    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Aug 04 17:59:48.863986 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:48.863968    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Aug 04 17:59:48.868126 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:48.868110    2583 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-n8n2f"
Aug 04 17:59:48.878733 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:48.878713    2583 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-n8n2f"
Aug 04 17:59:48.930619 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:48.930591    2583 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Aug 04 17:59:48.941028 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:48.941003    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/83f26afd-f898-4d64-8d22-4be7a1f5d862-host-slash\") pod \"ovnkube-node-mzl4m\" (UID: \"83f26afd-f898-4d64-8d22-4be7a1f5d862\") " pod="openshift-ovn-kubernetes/ovnkube-node-mzl4m"
Aug 04 17:59:48.941138 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:48.941047    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a8dd7a27-922c-4dde-bead-839291637a79-iptables-alerter-script\") pod \"iptables-alerter-rtcrf\" (UID: \"a8dd7a27-922c-4dde-bead-839291637a79\") " pod="openshift-network-operator/iptables-alerter-rtcrf"
Aug 04 17:59:48.941138 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:48.941075    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/de498784-c90b-4e01-bab8-e2ba98d49466-etc-kubernetes\") pod \"tuned-7bwz5\" (UID: \"de498784-c90b-4e01-bab8-e2ba98d49466\") " pod="openshift-cluster-node-tuning-operator/tuned-7bwz5"
Aug 04 17:59:48.941138 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:48.941102    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/de498784-c90b-4e01-bab8-e2ba98d49466-etc-sysctl-d\") pod \"tuned-7bwz5\" (UID: \"de498784-c90b-4e01-bab8-e2ba98d49466\") " pod="openshift-cluster-node-tuning-operator/tuned-7bwz5"
Aug 04 17:59:48.941276 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:48.941126    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3145f534-869a-4d4f-b69e-628803f76630-etc-kubernetes\") pod \"multus-jwcng\" (UID: \"3145f534-869a-4d4f-b69e-628803f76630\") " pod="openshift-multus/multus-jwcng"
Aug 04 17:59:48.941276 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:48.941167    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5f4f37d6-a72d-4707-a626-794c55098e25-kubelet-dir\") pod \"aws-ebs-csi-driver-node-57zmj\" (UID: \"5f4f37d6-a72d-4707-a626-794c55098e25\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-57zmj"
Aug 04 17:59:48.941276 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:48.941193    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kj22l\" (UniqueName: \"kubernetes.io/projected/a8dd7a27-922c-4dde-bead-839291637a79-kube-api-access-kj22l\") pod \"iptables-alerter-rtcrf\" (UID: \"a8dd7a27-922c-4dde-bead-839291637a79\") " pod="openshift-network-operator/iptables-alerter-rtcrf"
Aug 04 17:59:48.941276 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:48.941214    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/de498784-c90b-4e01-bab8-e2ba98d49466-etc-kubernetes\") pod \"tuned-7bwz5\" (UID: \"de498784-c90b-4e01-bab8-e2ba98d49466\") " pod="openshift-cluster-node-tuning-operator/tuned-7bwz5"
Aug 04 17:59:48.941409 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:48.941365    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/de498784-c90b-4e01-bab8-e2ba98d49466-etc-sysctl-d\") pod \"tuned-7bwz5\" (UID: \"de498784-c90b-4e01-bab8-e2ba98d49466\") " pod="openshift-cluster-node-tuning-operator/tuned-7bwz5"
Aug 04 17:59:48.941461 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:48.941219    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87rpt\" (UniqueName: \"kubernetes.io/projected/43697de6-c2bd-48c6-81eb-1f37fa5d3174-kube-api-access-87rpt\") pod \"network-check-target-jxnpj\" (UID: \"43697de6-c2bd-48c6-81eb-1f37fa5d3174\") " pod="openshift-network-diagnostics/network-check-target-jxnpj"
Aug 04 17:59:48.941535 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:48.941484    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3145f534-869a-4d4f-b69e-628803f76630-host-var-lib-kubelet\") pod \"multus-jwcng\" (UID: \"3145f534-869a-4d4f-b69e-628803f76630\") " pod="openshift-multus/multus-jwcng"
Aug 04 17:59:48.941535 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:48.941527    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/83f26afd-f898-4d64-8d22-4be7a1f5d862-run-ovn\") pod \"ovnkube-node-mzl4m\" (UID: \"83f26afd-f898-4d64-8d22-4be7a1f5d862\") " pod="openshift-ovn-kubernetes/ovnkube-node-mzl4m"
Aug 04 17:59:48.941640 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:48.941547    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/83f26afd-f898-4d64-8d22-4be7a1f5d862-ovn-node-metrics-cert\") pod \"ovnkube-node-mzl4m\" (UID: \"83f26afd-f898-4d64-8d22-4be7a1f5d862\") " pod="openshift-ovn-kubernetes/ovnkube-node-mzl4m"
Aug 04 17:59:48.941640 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:48.941565    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/555d14c0-e9e6-4119-8298-a88258e7c516-system-cni-dir\") pod \"multus-additional-cni-plugins-c6m8x\" (UID: \"555d14c0-e9e6-4119-8298-a88258e7c516\") " pod="openshift-multus/multus-additional-cni-plugins-c6m8x"
Aug 04 17:59:48.941640 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:48.941582    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znbjv\" (UniqueName: \"kubernetes.io/projected/555d14c0-e9e6-4119-8298-a88258e7c516-kube-api-access-znbjv\") pod \"multus-additional-cni-plugins-c6m8x\" (UID: \"555d14c0-e9e6-4119-8298-a88258e7c516\") " pod="openshift-multus/multus-additional-cni-plugins-c6m8x"
Aug 04 17:59:48.941640 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:48.941602    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/de498784-c90b-4e01-bab8-e2ba98d49466-tmp\") pod \"tuned-7bwz5\" (UID: \"de498784-c90b-4e01-bab8-e2ba98d49466\") " pod="openshift-cluster-node-tuning-operator/tuned-7bwz5"
Aug 04 17:59:48.941640 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:48.941618    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5f4f37d6-a72d-4707-a626-794c55098e25-socket-dir\") pod \"aws-ebs-csi-driver-node-57zmj\" (UID: \"5f4f37d6-a72d-4707-a626-794c55098e25\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-57zmj"
Aug 04 17:59:48.941640 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:48.941634    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/83f26afd-f898-4d64-8d22-4be7a1f5d862-host-kubelet\") pod \"ovnkube-node-mzl4m\" (UID: \"83f26afd-f898-4d64-8d22-4be7a1f5d862\") " pod="openshift-ovn-kubernetes/ovnkube-node-mzl4m"
Aug 04 17:59:48.941918 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:48.941648    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/83f26afd-f898-4d64-8d22-4be7a1f5d862-node-log\") pod \"ovnkube-node-mzl4m\" (UID: \"83f26afd-f898-4d64-8d22-4be7a1f5d862\") " pod="openshift-ovn-kubernetes/ovnkube-node-mzl4m"
Aug 04 17:59:48.941918 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:48.941662    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/83f26afd-f898-4d64-8d22-4be7a1f5d862-log-socket\") pod \"ovnkube-node-mzl4m\" (UID: \"83f26afd-f898-4d64-8d22-4be7a1f5d862\") " pod="openshift-ovn-kubernetes/ovnkube-node-mzl4m"
Aug 04 17:59:48.941918 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:48.941684    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/555d14c0-e9e6-4119-8298-a88258e7c516-os-release\") pod \"multus-additional-cni-plugins-c6m8x\" (UID: \"555d14c0-e9e6-4119-8298-a88258e7c516\") " pod="openshift-multus/multus-additional-cni-plugins-c6m8x"
Aug 04 17:59:48.941918 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:48.941696    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a8dd7a27-922c-4dde-bead-839291637a79-iptables-alerter-script\") pod \"iptables-alerter-rtcrf\" (UID: \"a8dd7a27-922c-4dde-bead-839291637a79\") " pod="openshift-network-operator/iptables-alerter-rtcrf"
Aug 04 17:59:48.941918 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:48.941701    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a8dd7a27-922c-4dde-bead-839291637a79-host-slash\") pod \"iptables-alerter-rtcrf\" (UID: \"a8dd7a27-922c-4dde-bead-839291637a79\") " pod="openshift-network-operator/iptables-alerter-rtcrf"
Aug 04 17:59:48.941918 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:48.941746    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a8dd7a27-922c-4dde-bead-839291637a79-host-slash\") pod \"iptables-alerter-rtcrf\" (UID: \"a8dd7a27-922c-4dde-bead-839291637a79\") " pod="openshift-network-operator/iptables-alerter-rtcrf"
Aug 04 17:59:48.941918 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:48.941764    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/de498784-c90b-4e01-bab8-e2ba98d49466-lib-modules\") pod \"tuned-7bwz5\" (UID: \"de498784-c90b-4e01-bab8-e2ba98d49466\") " pod="openshift-cluster-node-tuning-operator/tuned-7bwz5"
Aug 04 17:59:48.941918 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:48.941844    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/87245716-80f6-43f4-8f06-a0f3924615b2-agent-certs\") pod \"konnectivity-agent-ckl6v\" (UID: \"87245716-80f6-43f4-8f06-a0f3924615b2\") " pod="kube-system/konnectivity-agent-ckl6v"
Aug 04 17:59:48.941918 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:48.941878    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3145f534-869a-4d4f-b69e-628803f76630-cnibin\") pod \"multus-jwcng\" (UID: \"3145f534-869a-4d4f-b69e-628803f76630\") " pod="openshift-multus/multus-jwcng"
Aug 04 17:59:48.941918 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:48.941897    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/de498784-c90b-4e01-bab8-e2ba98d49466-lib-modules\") pod \"tuned-7bwz5\" (UID: \"de498784-c90b-4e01-bab8-e2ba98d49466\") " pod="openshift-cluster-node-tuning-operator/tuned-7bwz5"
Aug 04 17:59:48.941918 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:48.941916    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3145f534-869a-4d4f-b69e-628803f76630-cni-binary-copy\") pod \"multus-jwcng\" (UID: \"3145f534-869a-4d4f-b69e-628803f76630\") " pod="openshift-multus/multus-jwcng"
Aug 04 17:59:48.942422 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:48.941947    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/83f26afd-f898-4d64-8d22-4be7a1f5d862-host-cni-netd\") pod \"ovnkube-node-mzl4m\" (UID: \"83f26afd-f898-4d64-8d22-4be7a1f5d862\") " pod="openshift-ovn-kubernetes/ovnkube-node-mzl4m"
Aug 04 17:59:48.942422 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:48.941969    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzdfk\" (UniqueName: \"kubernetes.io/projected/83f26afd-f898-4d64-8d22-4be7a1f5d862-kube-api-access-gzdfk\") pod \"ovnkube-node-mzl4m\" (UID: \"83f26afd-f898-4d64-8d22-4be7a1f5d862\") " pod="openshift-ovn-kubernetes/ovnkube-node-mzl4m"
Aug 04 17:59:48.942422 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:48.942018    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9fffa81f-5a52-4425-b941-1693ba961fba-hosts-file\") pod \"node-resolver-ggrqd\" (UID: \"9fffa81f-5a52-4425-b941-1693ba961fba\") " pod="openshift-dns/node-resolver-ggrqd"
Aug 04 17:59:48.942422 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:48.942055    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3145f534-869a-4d4f-b69e-628803f76630-system-cni-dir\") pod \"multus-jwcng\" (UID: \"3145f534-869a-4d4f-b69e-628803f76630\") " pod="openshift-multus/multus-jwcng"
Aug 04 17:59:48.942422 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:48.942067    2583 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Aug 04 17:59:48.942422 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:48.942083    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3145f534-869a-4d4f-b69e-628803f76630-multus-socket-dir-parent\") pod \"multus-jwcng\" (UID: \"3145f534-869a-4d4f-b69e-628803f76630\") " pod="openshift-multus/multus-jwcng"
Aug 04 17:59:48.942422 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:48.942111    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3145f534-869a-4d4f-b69e-628803f76630-multus-daemon-config\") pod \"multus-jwcng\" (UID: \"3145f534-869a-4d4f-b69e-628803f76630\") " pod="openshift-multus/multus-jwcng"
Aug 04 17:59:48.942422 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:48.942135    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/00bf4681-2beb-4489-aba4-7e85f7bf2aaa-serviceca\") pod \"node-ca-nz5rh\" (UID: \"00bf4681-2beb-4489-aba4-7e85f7bf2aaa\") " pod="openshift-image-registry/node-ca-nz5rh"
Aug 04 17:59:48.942422 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:48.942171    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxd96\" (UniqueName: \"kubernetes.io/projected/00bf4681-2beb-4489-aba4-7e85f7bf2aaa-kube-api-access-fxd96\") pod \"node-ca-nz5rh\" (UID: \"00bf4681-2beb-4489-aba4-7e85f7bf2aaa\") " pod="openshift-image-registry/node-ca-nz5rh"
Aug 04 17:59:48.942422 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:48.942194    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5f4f37d6-a72d-4707-a626-794c55098e25-sys-fs\") pod \"aws-ebs-csi-driver-node-57zmj\" (UID: \"5f4f37d6-a72d-4707-a626-794c55098e25\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-57zmj"
Aug 04 17:59:48.942422 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:48.942222    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/83f26afd-f898-4d64-8d22-4be7a1f5d862-systemd-units\") pod \"ovnkube-node-mzl4m\" (UID: \"83f26afd-f898-4d64-8d22-4be7a1f5d862\") " pod="openshift-ovn-kubernetes/ovnkube-node-mzl4m"
Aug 04 17:59:48.942422 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:48.942246    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkhmf\" (UniqueName: \"kubernetes.io/projected/9fffa81f-5a52-4425-b941-1693ba961fba-kube-api-access-kkhmf\") pod \"node-resolver-ggrqd\" (UID: \"9fffa81f-5a52-4425-b941-1693ba961fba\") " pod="openshift-dns/node-resolver-ggrqd"
Aug 04 17:59:48.942422 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:48.942271    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3145f534-869a-4d4f-b69e-628803f76630-hostroot\") pod \"multus-jwcng\" (UID: \"3145f534-869a-4d4f-b69e-628803f76630\") " pod="openshift-multus/multus-jwcng"
Aug 04 17:59:48.942422 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:48.942290    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/83f26afd-f898-4d64-8d22-4be7a1f5d862-etc-openvswitch\") pod \"ovnkube-node-mzl4m\" (UID: \"83f26afd-f898-4d64-8d22-4be7a1f5d862\") " pod="openshift-ovn-kubernetes/ovnkube-node-mzl4m"
Aug 04 17:59:48.942422 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:48.942315    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/31fd88d9-8ce3-4056-a185-4427ec8939c1-original-pull-secret\") pod \"global-pull-secret-syncer-lqxxl\" (UID: \"31fd88d9-8ce3-4056-a185-4427ec8939c1\") " pod="kube-system/global-pull-secret-syncer-lqxxl"
Aug 04 17:59:48.942422 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:48.942338    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/de498784-c90b-4e01-bab8-e2ba98d49466-host\") pod \"tuned-7bwz5\" (UID: \"de498784-c90b-4e01-bab8-e2ba98d49466\") " pod="openshift-cluster-node-tuning-operator/tuned-7bwz5"
Aug 04 17:59:48.942422 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:48.942358    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4v564\" (UniqueName: \"kubernetes.io/projected/de498784-c90b-4e01-bab8-e2ba98d49466-kube-api-access-4v564\") pod \"tuned-7bwz5\" (UID: \"de498784-c90b-4e01-bab8-e2ba98d49466\") " pod="openshift-cluster-node-tuning-operator/tuned-7bwz5"
Aug 04 17:59:48.943316 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:48.942380    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/83f26afd-f898-4d64-8d22-4be7a1f5d862-var-lib-openvswitch\") pod \"ovnkube-node-mzl4m\" (UID: \"83f26afd-f898-4d64-8d22-4be7a1f5d862\") " pod="openshift-ovn-kubernetes/ovnkube-node-mzl4m"
Aug 04 17:59:48.943316 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:48.942405    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/83f26afd-f898-4d64-8d22-4be7a1f5d862-run-openvswitch\") pod \"ovnkube-node-mzl4m\" (UID: \"83f26afd-f898-4d64-8d22-4be7a1f5d862\") " pod="openshift-ovn-kubernetes/ovnkube-node-mzl4m"
Aug 04 17:59:48.943316 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:48.942421    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/de498784-c90b-4e01-bab8-e2ba98d49466-host\") pod \"tuned-7bwz5\" (UID: \"de498784-c90b-4e01-bab8-e2ba98d49466\") " pod="openshift-cluster-node-tuning-operator/tuned-7bwz5"
Aug 04 17:59:48.943316 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:48.942430    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/de498784-c90b-4e01-bab8-e2ba98d49466-etc-modprobe-d\") pod \"tuned-7bwz5\" (UID: \"de498784-c90b-4e01-bab8-e2ba98d49466\") " pod="openshift-cluster-node-tuning-operator/tuned-7bwz5"
Aug 04 17:59:48.943316 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:48.942466    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/31fd88d9-8ce3-4056-a185-4427ec8939c1-kubelet-config\") pod \"global-pull-secret-syncer-lqxxl\" (UID: \"31fd88d9-8ce3-4056-a185-4427ec8939c1\") " pod="kube-system/global-pull-secret-syncer-lqxxl"
Aug 04 17:59:48.943316 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:48.942507    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3145f534-869a-4d4f-b69e-628803f76630-multus-conf-dir\") pod \"multus-jwcng\" (UID: \"3145f534-869a-4d4f-b69e-628803f76630\") " pod="openshift-multus/multus-jwcng"
Aug 04 17:59:48.943316 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:48.942533    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/83f26afd-f898-4d64-8d22-4be7a1f5d862-host-run-netns\") pod \"ovnkube-node-mzl4m\" (UID: \"83f26afd-f898-4d64-8d22-4be7a1f5d862\") " pod="openshift-ovn-kubernetes/ovnkube-node-mzl4m"
Aug 04 17:59:48.943316 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:48.942544    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/de498784-c90b-4e01-bab8-e2ba98d49466-etc-modprobe-d\") pod \"tuned-7bwz5\" (UID: \"de498784-c90b-4e01-bab8-e2ba98d49466\") " pod="openshift-cluster-node-tuning-operator/tuned-7bwz5"
Aug 04 17:59:48.943316 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:48.942561    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/de498784-c90b-4e01-bab8-e2ba98d49466-var-lib-kubelet\") pod \"tuned-7bwz5\" (UID: \"de498784-c90b-4e01-bab8-e2ba98d49466\") " pod="openshift-cluster-node-tuning-operator/tuned-7bwz5"
Aug 04 17:59:48.943316 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:48.942585    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/de498784-c90b-4e01-bab8-e2ba98d49466-etc-tuned\") pod \"tuned-7bwz5\" (UID: \"de498784-c90b-4e01-bab8-e2ba98d49466\") " pod="openshift-cluster-node-tuning-operator/tuned-7bwz5"
Aug 04 17:59:48.943316 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:48.942613    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/83f26afd-f898-4d64-8d22-4be7a1f5d862-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mzl4m\" (UID: \"83f26afd-f898-4d64-8d22-4be7a1f5d862\") " pod="openshift-ovn-kubernetes/ovnkube-node-mzl4m"
Aug 04 17:59:48.943316 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:48.942627    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/de498784-c90b-4e01-bab8-e2ba98d49466-var-lib-kubelet\") pod \"tuned-7bwz5\" (UID: \"de498784-c90b-4e01-bab8-e2ba98d49466\") " pod="openshift-cluster-node-tuning-operator/tuned-7bwz5"
Aug 04 17:59:48.943316 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:48.942641    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9fffa81f-5a52-4425-b941-1693ba961fba-tmp-dir\") pod \"node-resolver-ggrqd\" (UID: \"9fffa81f-5a52-4425-b941-1693ba961fba\") " pod="openshift-dns/node-resolver-ggrqd"
Aug 04 17:59:48.943316 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:48.942665    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/31fd88d9-8ce3-4056-a185-4427ec8939c1-dbus\") pod \"global-pull-secret-syncer-lqxxl\" (UID: \"31fd88d9-8ce3-4056-a185-4427ec8939c1\") " pod="kube-system/global-pull-secret-syncer-lqxxl"
Aug 04 17:59:48.943316 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:48.942690    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3145f534-869a-4d4f-b69e-628803f76630-multus-cni-dir\") pod \"multus-jwcng\" (UID: \"3145f534-869a-4d4f-b69e-628803f76630\") " pod="openshift-multus/multus-jwcng"
Aug 04 17:59:48.943316 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:48.942714    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/3145f534-869a-4d4f-b69e-628803f76630-os-release\") pod \"multus-jwcng\" (UID: \"3145f534-869a-4d4f-b69e-628803f76630\") " pod="openshift-multus/multus-jwcng"
Aug 04 17:59:48.943316 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:48.942736    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5f4f37d6-a72d-4707-a626-794c55098e25-registration-dir\") pod \"aws-ebs-csi-driver-node-57zmj\" (UID: \"5f4f37d6-a72d-4707-a626-794c55098e25\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-57zmj"
Aug 04 17:59:48.944129 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:48.942760    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/83f26afd-f898-4d64-8d22-4be7a1f5d862-env-overrides\") pod \"ovnkube-node-mzl4m\" (UID: \"83f26afd-f898-4d64-8d22-4be7a1f5d862\") " pod="openshift-ovn-kubernetes/ovnkube-node-mzl4m"
Aug 04 17:59:48.944129 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:48.942785    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/555d14c0-e9e6-4119-8298-a88258e7c516-tuning-conf-dir\") pod \"multus-additional-cni-plugins-c6m8x\" (UID: \"555d14c0-e9e6-4119-8298-a88258e7c516\") " pod="openshift-multus/multus-additional-cni-plugins-c6m8x"
Aug 04 17:59:48.944129 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:48.942842    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/555d14c0-e9e6-4119-8298-a88258e7c516-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-c6m8x\" (UID: \"555d14c0-e9e6-4119-8298-a88258e7c516\") " pod="openshift-multus/multus-additional-cni-plugins-c6m8x"
Aug 04 17:59:48.944129 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:48.942869    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/de498784-c90b-4e01-bab8-e2ba98d49466-etc-systemd\") pod \"tuned-7bwz5\" (UID: \"de498784-c90b-4e01-bab8-e2ba98d49466\") " pod="openshift-cluster-node-tuning-operator/tuned-7bwz5"
Aug 04 17:59:48.944129 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:48.942904    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3145f534-869a-4d4f-b69e-628803f76630-host-var-lib-cni-bin\") pod \"multus-jwcng\" (UID: \"3145f534-869a-4d4f-b69e-628803f76630\") " pod="openshift-multus/multus-jwcng"
Aug 04 17:59:48.944129 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:48.942928    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3145f534-869a-4d4f-b69e-628803f76630-host-var-lib-cni-multus\") pod \"multus-jwcng\" (UID: \"3145f534-869a-4d4f-b69e-628803f76630\") " pod="openshift-multus/multus-jwcng"
Aug 04 17:59:48.944129 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:48.942967    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5f4f37d6-a72d-4707-a626-794c55098e25-etc-selinux\") pod \"aws-ebs-csi-driver-node-57zmj\" (UID: \"5f4f37d6-a72d-4707-a626-794c55098e25\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-57zmj"
Aug 04 17:59:48.944129 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:48.942986    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/de498784-c90b-4e01-bab8-e2ba98d49466-etc-systemd\") pod \"tuned-7bwz5\" (UID: \"de498784-c90b-4e01-bab8-e2ba98d49466\") " pod="openshift-cluster-node-tuning-operator/tuned-7bwz5"
Aug 04 17:59:48.944129 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:48.942994    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/555d14c0-e9e6-4119-8298-a88258e7c516-cni-binary-copy\") pod \"multus-additional-cni-plugins-c6m8x\" (UID: \"555d14c0-e9e6-4119-8298-a88258e7c516\") " pod="openshift-multus/multus-additional-cni-plugins-c6m8x"
Aug 04 17:59:48.944129 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:48.943027    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/de498784-c90b-4e01-bab8-e2ba98d49466-etc-sysctl-conf\") pod \"tuned-7bwz5\" (UID: \"de498784-c90b-4e01-bab8-e2ba98d49466\") " pod="openshift-cluster-node-tuning-operator/tuned-7bwz5"
Aug 04 17:59:48.944129 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:48.943047    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/de498784-c90b-4e01-bab8-e2ba98d49466-run\") pod \"tuned-7bwz5\" (UID: \"de498784-c90b-4e01-bab8-e2ba98d49466\") " pod="openshift-cluster-node-tuning-operator/tuned-7bwz5"
Aug 04 17:59:48.944129 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:48.943065    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3145f534-869a-4d4f-b69e-628803f76630-host-run-k8s-cni-cncf-io\") pod \"multus-jwcng\" (UID: \"3145f534-869a-4d4f-b69e-628803f76630\") " pod="openshift-multus/multus-jwcng"
Aug 04 17:59:48.944129 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:48.943086    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3145f534-869a-4d4f-b69e-628803f76630-host-run-multus-certs\") pod \"multus-jwcng\" (UID: \"3145f534-869a-4d4f-b69e-628803f76630\") " pod="openshift-multus/multus-jwcng"
Aug 04 17:59:48.944129 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:48.943110    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/83f26afd-f898-4d64-8d22-4be7a1f5d862-host-run-ovn-kubernetes\") pod \"ovnkube-node-mzl4m\" (UID: \"83f26afd-f898-4d64-8d22-4be7a1f5d862\") " pod="openshift-ovn-kubernetes/ovnkube-node-mzl4m"
Aug 04 17:59:48.944129 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:48.943133    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/83f26afd-f898-4d64-8d22-4be7a1f5d862-ovnkube-config\") pod \"ovnkube-node-mzl4m\" (UID: \"83f26afd-f898-4d64-8d22-4be7a1f5d862\") " pod="openshift-ovn-kubernetes/ovnkube-node-mzl4m"
Aug 04 17:59:48.944129 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:48.943171    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/83f26afd-f898-4d64-8d22-4be7a1f5d862-ovnkube-script-lib\") pod \"ovnkube-node-mzl4m\" (UID: \"83f26afd-f898-4d64-8d22-4be7a1f5d862\") " pod="openshift-ovn-kubernetes/ovnkube-node-mzl4m"
Aug 04 17:59:48.945003 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:48.943173    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/de498784-c90b-4e01-bab8-e2ba98d49466-etc-sysctl-conf\") pod \"tuned-7bwz5\" (UID: \"de498784-c90b-4e01-bab8-e2ba98d49466\") " pod="openshift-cluster-node-tuning-operator/tuned-7bwz5"
Aug 04 17:59:48.945003 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:48.943186    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/de498784-c90b-4e01-bab8-e2ba98d49466-run\") pod \"tuned-7bwz5\" (UID: \"de498784-c90b-4e01-bab8-e2ba98d49466\") " pod="openshift-cluster-node-tuning-operator/tuned-7bwz5"
Aug 04 17:59:48.945003 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:48.943196    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5f4f37d6-a72d-4707-a626-794c55098e25-device-dir\") pod \"aws-ebs-csi-driver-node-57zmj\" (UID: \"5f4f37d6-a72d-4707-a626-794c55098e25\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-57zmj"
Aug 04 17:59:48.945003 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:48.943222    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4kzk\" (UniqueName: \"kubernetes.io/projected/5f4f37d6-a72d-4707-a626-794c55098e25-kube-api-access-r4kzk\") pod \"aws-ebs-csi-driver-node-57zmj\" (UID: \"5f4f37d6-a72d-4707-a626-794c55098e25\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-57zmj"
Aug 04 17:59:48.945003 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:48.943245    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/83f26afd-f898-4d64-8d22-4be7a1f5d862-host-cni-bin\") pod \"ovnkube-node-mzl4m\" (UID: \"83f26afd-f898-4d64-8d22-4be7a1f5d862\") " pod="openshift-ovn-kubernetes/ovnkube-node-mzl4m"
Aug 04 17:59:48.945003 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:48.943269    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/555d14c0-e9e6-4119-8298-a88258e7c516-cnibin\") pod \"multus-additional-cni-plugins-c6m8x\" (UID: \"555d14c0-e9e6-4119-8298-a88258e7c516\") " pod="openshift-multus/multus-additional-cni-plugins-c6m8x"
Aug 04 17:59:48.945003 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:48.943292    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/de498784-c90b-4e01-bab8-e2ba98d49466-etc-sysconfig\") pod \"tuned-7bwz5\" (UID: \"de498784-c90b-4e01-bab8-e2ba98d49466\") " pod="openshift-cluster-node-tuning-operator/tuned-7bwz5"
Aug 04 17:59:48.945003 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:48.943342    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/de498784-c90b-4e01-bab8-e2ba98d49466-sys\") pod \"tuned-7bwz5\" (UID: \"de498784-c90b-4e01-bab8-e2ba98d49466\") " pod="openshift-cluster-node-tuning-operator/tuned-7bwz5"
Aug 04 17:59:48.945003 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:48.943369    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4pkwv\" (UniqueName: \"kubernetes.io/projected/0480eb75-3ca1-46ba-9606-ec9e98dc9c4c-kube-api-access-4pkwv\") pod \"network-metrics-daemon-qw2tv\" (UID: \"0480eb75-3ca1-46ba-9606-ec9e98dc9c4c\") " pod="openshift-multus/network-metrics-daemon-qw2tv"
Aug 04 17:59:48.945003 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:48.943402    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/de498784-c90b-4e01-bab8-e2ba98d49466-etc-sysconfig\") pod \"tuned-7bwz5\" (UID: \"de498784-c90b-4e01-bab8-e2ba98d49466\") " pod="openshift-cluster-node-tuning-operator/tuned-7bwz5"
Aug 04 17:59:48.945003 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:48.943457    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/de498784-c90b-4e01-bab8-e2ba98d49466-sys\") pod \"tuned-7bwz5\" (UID: \"de498784-c90b-4e01-bab8-e2ba98d49466\") " pod="openshift-cluster-node-tuning-operator/tuned-7bwz5"
Aug 04 17:59:48.945003 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:48.943490    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/00bf4681-2beb-4489-aba4-7e85f7bf2aaa-host\") pod \"node-ca-nz5rh\" (UID: \"00bf4681-2beb-4489-aba4-7e85f7bf2aaa\") " pod="openshift-image-registry/node-ca-nz5rh"
Aug 04 17:59:48.945003 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:48.943536    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/83f26afd-f898-4d64-8d22-4be7a1f5d862-run-systemd\") pod \"ovnkube-node-mzl4m\" (UID: \"83f26afd-f898-4d64-8d22-4be7a1f5d862\") " pod="openshift-ovn-kubernetes/ovnkube-node-mzl4m"
Aug 04 17:59:48.945003 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:48.943565    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/555d14c0-e9e6-4119-8298-a88258e7c516-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-c6m8x\" (UID: \"555d14c0-e9e6-4119-8298-a88258e7c516\") " pod="openshift-multus/multus-additional-cni-plugins-c6m8x"
Aug 04 17:59:48.945003 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:48.943593    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0480eb75-3ca1-46ba-9606-ec9e98dc9c4c-metrics-certs\") pod \"network-metrics-daemon-qw2tv\" (UID: \"0480eb75-3ca1-46ba-9606-ec9e98dc9c4c\") " pod="openshift-multus/network-metrics-daemon-qw2tv"
Aug 04 17:59:48.945003 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:48.943618    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/87245716-80f6-43f4-8f06-a0f3924615b2-konnectivity-ca\") pod \"konnectivity-agent-ckl6v\" (UID: \"87245716-80f6-43f4-8f06-a0f3924615b2\") " pod="kube-system/konnectivity-agent-ckl6v"
Aug 04 17:59:48.945003 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:48.943640    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3145f534-869a-4d4f-b69e-628803f76630-host-run-netns\") pod \"multus-jwcng\" (UID: \"3145f534-869a-4d4f-b69e-628803f76630\") " pod="openshift-multus/multus-jwcng"
Aug 04 17:59:48.945911 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:48.943663    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xd2jf\" (UniqueName: \"kubernetes.io/projected/3145f534-869a-4d4f-b69e-628803f76630-kube-api-access-xd2jf\") pod \"multus-jwcng\" (UID: \"3145f534-869a-4d4f-b69e-628803f76630\") " pod="openshift-multus/multus-jwcng"
Aug 04 17:59:48.945911 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:48.943968    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-124.ec2.internal" event={"ID":"f727e1c61c596ab926797040f56320d1","Type":"ContainerStarted","Data":"2c2009093336b04ede0f138e853ce2f03d6a96be292e1b1fb787d686ab0dbc94"}
Aug 04 17:59:48.946723 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:48.946672    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-124.ec2.internal" event={"ID":"4e7e6ce51e0af982f917261a0b47d870","Type":"ContainerStarted","Data":"17a7cf982218c7816a2d6a94653437ad4a525d8b01ccf7d05a1bb6123a07383f"}
Aug 04 17:59:48.946866 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:48.946845    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/de498784-c90b-4e01-bab8-e2ba98d49466-tmp\") pod \"tuned-7bwz5\" (UID: \"de498784-c90b-4e01-bab8-e2ba98d49466\") " pod="openshift-cluster-node-tuning-operator/tuned-7bwz5"
Aug 04 17:59:48.947141 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:48.947119    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/de498784-c90b-4e01-bab8-e2ba98d49466-etc-tuned\") pod \"tuned-7bwz5\" (UID: \"de498784-c90b-4e01-bab8-e2ba98d49466\") " pod="openshift-cluster-node-tuning-operator/tuned-7bwz5"
Aug 04 17:59:48.947704 ip-10-0-135-124 kubenswrapper[2583]: E0804 17:59:48.947601    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Aug 04 17:59:48.947704 ip-10-0-135-124 kubenswrapper[2583]: E0804 17:59:48.947643    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Aug 04 17:59:48.947704 ip-10-0-135-124 kubenswrapper[2583]: E0804 17:59:48.947658    2583 projected.go:196] Error preparing data for projected volume kube-api-access-87rpt for pod openshift-network-diagnostics/network-check-target-jxnpj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 04 17:59:48.948326 ip-10-0-135-124 kubenswrapper[2583]: E0804 17:59:48.947753    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/43697de6-c2bd-48c6-81eb-1f37fa5d3174-kube-api-access-87rpt podName:43697de6-c2bd-48c6-81eb-1f37fa5d3174 nodeName:}" failed. No retries permitted until 2026-08-04 17:59:49.447729948 +0000 UTC m=+3.028948521 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-87rpt" (UniqueName: "kubernetes.io/projected/43697de6-c2bd-48c6-81eb-1f37fa5d3174-kube-api-access-87rpt") pod "network-check-target-jxnpj" (UID: "43697de6-c2bd-48c6-81eb-1f37fa5d3174") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 04 17:59:48.950489 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:48.950464    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4v564\" (UniqueName: \"kubernetes.io/projected/de498784-c90b-4e01-bab8-e2ba98d49466-kube-api-access-4v564\") pod \"tuned-7bwz5\" (UID: \"de498784-c90b-4e01-bab8-e2ba98d49466\") " pod="openshift-cluster-node-tuning-operator/tuned-7bwz5"
Aug 04 17:59:48.950795 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:48.950777    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kj22l\" (UniqueName: \"kubernetes.io/projected/a8dd7a27-922c-4dde-bead-839291637a79-kube-api-access-kj22l\") pod \"iptables-alerter-rtcrf\" (UID: \"a8dd7a27-922c-4dde-bead-839291637a79\") " pod="openshift-network-operator/iptables-alerter-rtcrf"
Aug 04 17:59:49.044000 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:49.043966    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3145f534-869a-4d4f-b69e-628803f76630-host-var-lib-cni-bin\") pod \"multus-jwcng\" (UID: \"3145f534-869a-4d4f-b69e-628803f76630\") " pod="openshift-multus/multus-jwcng"
Aug 04 17:59:49.044185 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:49.044008    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3145f534-869a-4d4f-b69e-628803f76630-host-var-lib-cni-multus\") pod \"multus-jwcng\" (UID: \"3145f534-869a-4d4f-b69e-628803f76630\") " pod="openshift-multus/multus-jwcng"
Aug 04 17:59:49.044185 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:49.044033    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5f4f37d6-a72d-4707-a626-794c55098e25-etc-selinux\") pod \"aws-ebs-csi-driver-node-57zmj\" (UID: \"5f4f37d6-a72d-4707-a626-794c55098e25\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-57zmj"
Aug 04 17:59:49.044185 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:49.044058    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/555d14c0-e9e6-4119-8298-a88258e7c516-cni-binary-copy\") pod \"multus-additional-cni-plugins-c6m8x\" (UID: \"555d14c0-e9e6-4119-8298-a88258e7c516\") " pod="openshift-multus/multus-additional-cni-plugins-c6m8x"
Aug 04 17:59:49.044185 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:49.044066    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3145f534-869a-4d4f-b69e-628803f76630-host-var-lib-cni-multus\") pod \"multus-jwcng\" (UID: \"3145f534-869a-4d4f-b69e-628803f76630\") " pod="openshift-multus/multus-jwcng"
Aug 04 17:59:49.044185 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:49.044094    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3145f534-869a-4d4f-b69e-628803f76630-host-run-k8s-cni-cncf-io\") pod \"multus-jwcng\" (UID: \"3145f534-869a-4d4f-b69e-628803f76630\") " pod="openshift-multus/multus-jwcng"
Aug 04 17:59:49.044185 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:49.044082    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3145f534-869a-4d4f-b69e-628803f76630-host-var-lib-cni-bin\") pod \"multus-jwcng\" (UID: \"3145f534-869a-4d4f-b69e-628803f76630\") " pod="openshift-multus/multus-jwcng"
Aug 04 17:59:49.044185 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:49.044115    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5f4f37d6-a72d-4707-a626-794c55098e25-etc-selinux\") pod \"aws-ebs-csi-driver-node-57zmj\" (UID: \"5f4f37d6-a72d-4707-a626-794c55098e25\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-57zmj"
Aug 04 17:59:49.044185 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:49.044120    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3145f534-869a-4d4f-b69e-628803f76630-host-run-multus-certs\") pod \"multus-jwcng\" (UID: \"3145f534-869a-4d4f-b69e-628803f76630\") " pod="openshift-multus/multus-jwcng"
Aug 04 17:59:49.044185 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:49.044149    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3145f534-869a-4d4f-b69e-628803f76630-host-run-k8s-cni-cncf-io\") pod \"multus-jwcng\" (UID: \"3145f534-869a-4d4f-b69e-628803f76630\") " pod="openshift-multus/multus-jwcng"
Aug 04 17:59:49.044185 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:49.044178    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3145f534-869a-4d4f-b69e-628803f76630-host-run-multus-certs\") pod \"multus-jwcng\" (UID: \"3145f534-869a-4d4f-b69e-628803f76630\") " pod="openshift-multus/multus-jwcng"
Aug 04 17:59:49.044760 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:49.044211    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/83f26afd-f898-4d64-8d22-4be7a1f5d862-host-run-ovn-kubernetes\") pod \"ovnkube-node-mzl4m\" (UID: \"83f26afd-f898-4d64-8d22-4be7a1f5d862\") " pod="openshift-ovn-kubernetes/ovnkube-node-mzl4m"
Aug 04 17:59:49.044760 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:49.044299    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/83f26afd-f898-4d64-8d22-4be7a1f5d862-ovnkube-config\") pod \"ovnkube-node-mzl4m\" (UID: \"83f26afd-f898-4d64-8d22-4be7a1f5d862\") " pod="openshift-ovn-kubernetes/ovnkube-node-mzl4m"
Aug 04 17:59:49.044760 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:49.044331    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/83f26afd-f898-4d64-8d22-4be7a1f5d862-ovnkube-script-lib\") pod \"ovnkube-node-mzl4m\" (UID: \"83f26afd-f898-4d64-8d22-4be7a1f5d862\") " pod="openshift-ovn-kubernetes/ovnkube-node-mzl4m"
Aug 04 17:59:49.044760 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:49.044342    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/83f26afd-f898-4d64-8d22-4be7a1f5d862-host-run-ovn-kubernetes\") pod \"ovnkube-node-mzl4m\" (UID: \"83f26afd-f898-4d64-8d22-4be7a1f5d862\") " pod="openshift-ovn-kubernetes/ovnkube-node-mzl4m"
Aug 04 17:59:49.044760 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:49.044353    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5f4f37d6-a72d-4707-a626-794c55098e25-device-dir\") pod \"aws-ebs-csi-driver-node-57zmj\" (UID: \"5f4f37d6-a72d-4707-a626-794c55098e25\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-57zmj"
Aug 04 17:59:49.044760 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:49.044414    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4kzk\" (UniqueName: \"kubernetes.io/projected/5f4f37d6-a72d-4707-a626-794c55098e25-kube-api-access-r4kzk\") pod \"aws-ebs-csi-driver-node-57zmj\" (UID: \"5f4f37d6-a72d-4707-a626-794c55098e25\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-57zmj"
Aug 04 17:59:49.044760 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:49.044419    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5f4f37d6-a72d-4707-a626-794c55098e25-device-dir\") pod \"aws-ebs-csi-driver-node-57zmj\" (UID: \"5f4f37d6-a72d-4707-a626-794c55098e25\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-57zmj"
Aug 04 17:59:49.044760 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:49.044533    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/83f26afd-f898-4d64-8d22-4be7a1f5d862-host-cni-bin\") pod \"ovnkube-node-mzl4m\" (UID: \"83f26afd-f898-4d64-8d22-4be7a1f5d862\") " pod="openshift-ovn-kubernetes/ovnkube-node-mzl4m"
Aug 04 17:59:49.044760 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:49.044568    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/555d14c0-e9e6-4119-8298-a88258e7c516-cnibin\") pod \"multus-additional-cni-plugins-c6m8x\" (UID: \"555d14c0-e9e6-4119-8298-a88258e7c516\") " pod="openshift-multus/multus-additional-cni-plugins-c6m8x"
Aug 04 17:59:49.044760 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:49.044599    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pkwv\" (UniqueName: \"kubernetes.io/projected/0480eb75-3ca1-46ba-9606-ec9e98dc9c4c-kube-api-access-4pkwv\") pod \"network-metrics-daemon-qw2tv\" (UID: \"0480eb75-3ca1-46ba-9606-ec9e98dc9c4c\") " pod="openshift-multus/network-metrics-daemon-qw2tv"
Aug 04 17:59:49.044760 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:49.044618    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/83f26afd-f898-4d64-8d22-4be7a1f5d862-host-cni-bin\") pod \"ovnkube-node-mzl4m\" (UID: \"83f26afd-f898-4d64-8d22-4be7a1f5d862\") " pod="openshift-ovn-kubernetes/ovnkube-node-mzl4m"
Aug 04 17:59:49.044760 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:49.044625    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/00bf4681-2beb-4489-aba4-7e85f7bf2aaa-host\") pod \"node-ca-nz5rh\" (UID: \"00bf4681-2beb-4489-aba4-7e85f7bf2aaa\") " pod="openshift-image-registry/node-ca-nz5rh"
Aug 04 17:59:49.044760 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:49.044674    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/00bf4681-2beb-4489-aba4-7e85f7bf2aaa-host\") pod \"node-ca-nz5rh\" (UID: \"00bf4681-2beb-4489-aba4-7e85f7bf2aaa\") " pod="openshift-image-registry/node-ca-nz5rh"
Aug 04 17:59:49.044760 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:49.044680    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/555d14c0-e9e6-4119-8298-a88258e7c516-cnibin\") pod \"multus-additional-cni-plugins-c6m8x\" (UID: \"555d14c0-e9e6-4119-8298-a88258e7c516\") " pod="openshift-multus/multus-additional-cni-plugins-c6m8x"
Aug 04 17:59:49.044760 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:49.044681    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/83f26afd-f898-4d64-8d22-4be7a1f5d862-run-systemd\") pod \"ovnkube-node-mzl4m\" (UID: \"83f26afd-f898-4d64-8d22-4be7a1f5d862\") " pod="openshift-ovn-kubernetes/ovnkube-node-mzl4m"
Aug 04 17:59:49.044760 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:49.044718    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/83f26afd-f898-4d64-8d22-4be7a1f5d862-run-systemd\") pod \"ovnkube-node-mzl4m\" (UID: \"83f26afd-f898-4d64-8d22-4be7a1f5d862\") " pod="openshift-ovn-kubernetes/ovnkube-node-mzl4m"
Aug 04 17:59:49.044760 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:49.044734    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/555d14c0-e9e6-4119-8298-a88258e7c516-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-c6m8x\" (UID: \"555d14c0-e9e6-4119-8298-a88258e7c516\") " pod="openshift-multus/multus-additional-cni-plugins-c6m8x"
Aug 04 17:59:49.045602 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:49.044760    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0480eb75-3ca1-46ba-9606-ec9e98dc9c4c-metrics-certs\") pod \"network-metrics-daemon-qw2tv\" (UID: \"0480eb75-3ca1-46ba-9606-ec9e98dc9c4c\") " pod="openshift-multus/network-metrics-daemon-qw2tv"
Aug 04 17:59:49.045602 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:49.044784    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/87245716-80f6-43f4-8f06-a0f3924615b2-konnectivity-ca\") pod \"konnectivity-agent-ckl6v\" (UID: \"87245716-80f6-43f4-8f06-a0f3924615b2\") " pod="kube-system/konnectivity-agent-ckl6v"
Aug 04 17:59:49.045602 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:49.044809    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3145f534-869a-4d4f-b69e-628803f76630-host-run-netns\") pod \"multus-jwcng\" (UID: \"3145f534-869a-4d4f-b69e-628803f76630\") " pod="openshift-multus/multus-jwcng"
Aug 04 17:59:49.045602 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:49.044834    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xd2jf\" (UniqueName: \"kubernetes.io/projected/3145f534-869a-4d4f-b69e-628803f76630-kube-api-access-xd2jf\") pod \"multus-jwcng\" (UID: \"3145f534-869a-4d4f-b69e-628803f76630\") " pod="openshift-multus/multus-jwcng"
Aug 04 17:59:49.045602 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:49.044860    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/83f26afd-f898-4d64-8d22-4be7a1f5d862-host-slash\") pod \"ovnkube-node-mzl4m\" (UID: \"83f26afd-f898-4d64-8d22-4be7a1f5d862\") " pod="openshift-ovn-kubernetes/ovnkube-node-mzl4m"
Aug 04 17:59:49.045602 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:49.044868    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/555d14c0-e9e6-4119-8298-a88258e7c516-cni-binary-copy\") pod \"multus-additional-cni-plugins-c6m8x\" (UID: \"555d14c0-e9e6-4119-8298-a88258e7c516\") " pod="openshift-multus/multus-additional-cni-plugins-c6m8x"
Aug 04 17:59:49.045602 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:49.044895    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3145f534-869a-4d4f-b69e-628803f76630-etc-kubernetes\") pod \"multus-jwcng\" (UID: \"3145f534-869a-4d4f-b69e-628803f76630\") " pod="openshift-multus/multus-jwcng"
Aug 04 17:59:49.045602 ip-10-0-135-124 kubenswrapper[2583]: E0804 17:59:49.044921    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 04 17:59:49.045602 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:49.044927    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5f4f37d6-a72d-4707-a626-794c55098e25-kubelet-dir\") pod \"aws-ebs-csi-driver-node-57zmj\" (UID: \"5f4f37d6-a72d-4707-a626-794c55098e25\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-57zmj"
Aug 04 17:59:49.045602 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:49.044979    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3145f534-869a-4d4f-b69e-628803f76630-host-var-lib-kubelet\") pod \"multus-jwcng\" (UID: \"3145f534-869a-4d4f-b69e-628803f76630\") " pod="openshift-multus/multus-jwcng"
Aug 04 17:59:49.045602 ip-10-0-135-124 kubenswrapper[2583]: E0804 17:59:49.045011    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0480eb75-3ca1-46ba-9606-ec9e98dc9c4c-metrics-certs podName:0480eb75-3ca1-46ba-9606-ec9e98dc9c4c nodeName:}" failed. No retries permitted until 2026-08-04 17:59:49.544991169 +0000 UTC m=+3.126209745 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0480eb75-3ca1-46ba-9606-ec9e98dc9c4c-metrics-certs") pod "network-metrics-daemon-qw2tv" (UID: "0480eb75-3ca1-46ba-9606-ec9e98dc9c4c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 04 17:59:49.045602 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:49.044923    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3145f534-869a-4d4f-b69e-628803f76630-host-run-netns\") pod \"multus-jwcng\" (UID: \"3145f534-869a-4d4f-b69e-628803f76630\") " pod="openshift-multus/multus-jwcng"
Aug 04 17:59:49.045602 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:49.045035    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3145f534-869a-4d4f-b69e-628803f76630-host-var-lib-kubelet\") pod \"multus-jwcng\" (UID: \"3145f534-869a-4d4f-b69e-628803f76630\") " pod="openshift-multus/multus-jwcng"
Aug 04 17:59:49.045602 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:49.045037    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/83f26afd-f898-4d64-8d22-4be7a1f5d862-run-ovn\") pod \"ovnkube-node-mzl4m\" (UID: \"83f26afd-f898-4d64-8d22-4be7a1f5d862\") " pod="openshift-ovn-kubernetes/ovnkube-node-mzl4m"
Aug 04 17:59:49.045602 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:49.045073    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/83f26afd-f898-4d64-8d22-4be7a1f5d862-ovn-node-metrics-cert\") pod \"ovnkube-node-mzl4m\" (UID: \"83f26afd-f898-4d64-8d22-4be7a1f5d862\") " pod="openshift-ovn-kubernetes/ovnkube-node-mzl4m"
Aug 04 17:59:49.045602 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:49.045077    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/83f26afd-f898-4d64-8d22-4be7a1f5d862-run-ovn\") pod \"ovnkube-node-mzl4m\" (UID: \"83f26afd-f898-4d64-8d22-4be7a1f5d862\") " pod="openshift-ovn-kubernetes/ovnkube-node-mzl4m"
Aug 04 17:59:49.045602 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:49.045009    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/83f26afd-f898-4d64-8d22-4be7a1f5d862-ovnkube-config\") pod \"ovnkube-node-mzl4m\" (UID: \"83f26afd-f898-4d64-8d22-4be7a1f5d862\") " pod="openshift-ovn-kubernetes/ovnkube-node-mzl4m"
Aug 04 17:59:49.046461 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:49.045153    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/83f26afd-f898-4d64-8d22-4be7a1f5d862-host-slash\") pod \"ovnkube-node-mzl4m\" (UID: \"83f26afd-f898-4d64-8d22-4be7a1f5d862\") " pod="openshift-ovn-kubernetes/ovnkube-node-mzl4m"
Aug 04 17:59:49.046461 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:49.045218    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3145f534-869a-4d4f-b69e-628803f76630-etc-kubernetes\") pod \"multus-jwcng\" (UID: \"3145f534-869a-4d4f-b69e-628803f76630\") " pod="openshift-multus/multus-jwcng"
Aug 04 17:59:49.046461 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:49.045236    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/83f26afd-f898-4d64-8d22-4be7a1f5d862-ovnkube-script-lib\") pod \"ovnkube-node-mzl4m\" (UID: \"83f26afd-f898-4d64-8d22-4be7a1f5d862\") " pod="openshift-ovn-kubernetes/ovnkube-node-mzl4m"
Aug 04 17:59:49.046461 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:49.045281    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/555d14c0-e9e6-4119-8298-a88258e7c516-system-cni-dir\") pod \"multus-additional-cni-plugins-c6m8x\" (UID: \"555d14c0-e9e6-4119-8298-a88258e7c516\") " pod="openshift-multus/multus-additional-cni-plugins-c6m8x"
Aug 04 17:59:49.046461 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:49.045314    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znbjv\" (UniqueName: \"kubernetes.io/projected/555d14c0-e9e6-4119-8298-a88258e7c516-kube-api-access-znbjv\") pod \"multus-additional-cni-plugins-c6m8x\" (UID: \"555d14c0-e9e6-4119-8298-a88258e7c516\") " pod="openshift-multus/multus-additional-cni-plugins-c6m8x"
Aug 04 17:59:49.046461 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:49.045346    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5f4f37d6-a72d-4707-a626-794c55098e25-socket-dir\") pod \"aws-ebs-csi-driver-node-57zmj\" (UID: \"5f4f37d6-a72d-4707-a626-794c55098e25\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-57zmj"
Aug 04 17:59:49.046461 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:49.045361    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/555d14c0-e9e6-4119-8298-a88258e7c516-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-c6m8x\" (UID: \"555d14c0-e9e6-4119-8298-a88258e7c516\") " pod="openshift-multus/multus-additional-cni-plugins-c6m8x"
Aug 04 17:59:49.046461 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:49.045371    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/83f26afd-f898-4d64-8d22-4be7a1f5d862-host-kubelet\") pod \"ovnkube-node-mzl4m\" (UID: \"83f26afd-f898-4d64-8d22-4be7a1f5d862\") " pod="openshift-ovn-kubernetes/ovnkube-node-mzl4m"
Aug 04 17:59:49.046461 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:49.045316    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/555d14c0-e9e6-4119-8298-a88258e7c516-system-cni-dir\") pod \"multus-additional-cni-plugins-c6m8x\" (UID: \"555d14c0-e9e6-4119-8298-a88258e7c516\") " pod="openshift-multus/multus-additional-cni-plugins-c6m8x"
Aug 04 17:59:49.046461 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:49.045397    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/83f26afd-f898-4d64-8d22-4be7a1f5d862-node-log\") pod \"ovnkube-node-mzl4m\" (UID: \"83f26afd-f898-4d64-8d22-4be7a1f5d862\") " pod="openshift-ovn-kubernetes/ovnkube-node-mzl4m"
Aug 04 17:59:49.046461 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:49.045282    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5f4f37d6-a72d-4707-a626-794c55098e25-kubelet-dir\") pod \"aws-ebs-csi-driver-node-57zmj\" (UID: \"5f4f37d6-a72d-4707-a626-794c55098e25\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-57zmj"
Aug 04 17:59:49.046461 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:49.045423    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/87245716-80f6-43f4-8f06-a0f3924615b2-konnectivity-ca\") pod \"konnectivity-agent-ckl6v\" (UID: \"87245716-80f6-43f4-8f06-a0f3924615b2\") " pod="kube-system/konnectivity-agent-ckl6v"
Aug 04 17:59:49.046461 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:49.045440    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/83f26afd-f898-4d64-8d22-4be7a1f5d862-host-kubelet\") pod \"ovnkube-node-mzl4m\" (UID: \"83f26afd-f898-4d64-8d22-4be7a1f5d862\") " pod="openshift-ovn-kubernetes/ovnkube-node-mzl4m"
Aug 04 17:59:49.046461 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:49.045448    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/83f26afd-f898-4d64-8d22-4be7a1f5d862-node-log\") pod \"ovnkube-node-mzl4m\" (UID: \"83f26afd-f898-4d64-8d22-4be7a1f5d862\") " pod="openshift-ovn-kubernetes/ovnkube-node-mzl4m"
Aug 04 17:59:49.046461 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:49.045460    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5f4f37d6-a72d-4707-a626-794c55098e25-socket-dir\") pod \"aws-ebs-csi-driver-node-57zmj\" (UID: \"5f4f37d6-a72d-4707-a626-794c55098e25\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-57zmj"
Aug 04 17:59:49.046461 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:49.045464    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/83f26afd-f898-4d64-8d22-4be7a1f5d862-log-socket\") pod \"ovnkube-node-mzl4m\" (UID: \"83f26afd-f898-4d64-8d22-4be7a1f5d862\") " pod="openshift-ovn-kubernetes/ovnkube-node-mzl4m"
Aug 04 17:59:49.046461 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:49.045504    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/83f26afd-f898-4d64-8d22-4be7a1f5d862-log-socket\") pod \"ovnkube-node-mzl4m\" (UID: \"83f26afd-f898-4d64-8d22-4be7a1f5d862\") " pod="openshift-ovn-kubernetes/ovnkube-node-mzl4m"
Aug 04 17:59:49.047331 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:49.045510    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/555d14c0-e9e6-4119-8298-a88258e7c516-os-release\") pod \"multus-additional-cni-plugins-c6m8x\" (UID: \"555d14c0-e9e6-4119-8298-a88258e7c516\") " pod="openshift-multus/multus-additional-cni-plugins-c6m8x"
Aug 04 17:59:49.047331 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:49.045637    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/555d14c0-e9e6-4119-8298-a88258e7c516-os-release\") pod \"multus-additional-cni-plugins-c6m8x\" (UID: \"555d14c0-e9e6-4119-8298-a88258e7c516\") " pod="openshift-multus/multus-additional-cni-plugins-c6m8x"
Aug 04 17:59:49.047331 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:49.045671    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/87245716-80f6-43f4-8f06-a0f3924615b2-agent-certs\") pod \"konnectivity-agent-ckl6v\" (UID: \"87245716-80f6-43f4-8f06-a0f3924615b2\") " pod="kube-system/konnectivity-agent-ckl6v"
Aug 04 17:59:49.047331 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:49.045690    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3145f534-869a-4d4f-b69e-628803f76630-cnibin\") pod \"multus-jwcng\" (UID: \"3145f534-869a-4d4f-b69e-628803f76630\") " pod="openshift-multus/multus-jwcng"
Aug 04 17:59:49.047331 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:49.045708    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3145f534-869a-4d4f-b69e-628803f76630-cni-binary-copy\") pod \"multus-jwcng\" (UID: \"3145f534-869a-4d4f-b69e-628803f76630\") " pod="openshift-multus/multus-jwcng"
Aug 04 17:59:49.047331 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:49.045734    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/83f26afd-f898-4d64-8d22-4be7a1f5d862-host-cni-netd\") pod \"ovnkube-node-mzl4m\" (UID: \"83f26afd-f898-4d64-8d22-4be7a1f5d862\") " pod="openshift-ovn-kubernetes/ovnkube-node-mzl4m"
Aug 04 17:59:49.047331 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:49.045760    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzdfk\" (UniqueName: \"kubernetes.io/projected/83f26afd-f898-4d64-8d22-4be7a1f5d862-kube-api-access-gzdfk\") pod \"ovnkube-node-mzl4m\" (UID: \"83f26afd-f898-4d64-8d22-4be7a1f5d862\") " pod="openshift-ovn-kubernetes/ovnkube-node-mzl4m"
Aug 04 17:59:49.047331 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:49.045787    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9fffa81f-5a52-4425-b941-1693ba961fba-hosts-file\") pod \"node-resolver-ggrqd\" (UID: \"9fffa81f-5a52-4425-b941-1693ba961fba\") " pod="openshift-dns/node-resolver-ggrqd"
Aug 04 17:59:49.047331 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:49.045793    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3145f534-869a-4d4f-b69e-628803f76630-cnibin\") pod \"multus-jwcng\" (UID: \"3145f534-869a-4d4f-b69e-628803f76630\") " pod="openshift-multus/multus-jwcng"
Aug 04 17:59:49.047331 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:49.045811    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3145f534-869a-4d4f-b69e-628803f76630-system-cni-dir\") pod \"multus-jwcng\" (UID: \"3145f534-869a-4d4f-b69e-628803f76630\") " pod="openshift-multus/multus-jwcng"
Aug 04 17:59:49.047331 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:49.045836    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3145f534-869a-4d4f-b69e-628803f76630-multus-socket-dir-parent\") pod \"multus-jwcng\" (UID: \"3145f534-869a-4d4f-b69e-628803f76630\") " pod="openshift-multus/multus-jwcng"
Aug 04 17:59:49.047331 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:49.045861    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3145f534-869a-4d4f-b69e-628803f76630-multus-daemon-config\") pod \"multus-jwcng\" (UID: \"3145f534-869a-4d4f-b69e-628803f76630\") " pod="openshift-multus/multus-jwcng"
Aug 04 17:59:49.047331 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:49.045883    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/00bf4681-2beb-4489-aba4-7e85f7bf2aaa-serviceca\") pod \"node-ca-nz5rh\" (UID: \"00bf4681-2beb-4489-aba4-7e85f7bf2aaa\") " pod="openshift-image-registry/node-ca-nz5rh"
Aug 04 17:59:49.047331 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:49.045906    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxd96\" (UniqueName: \"kubernetes.io/projected/00bf4681-2beb-4489-aba4-7e85f7bf2aaa-kube-api-access-fxd96\") pod \"node-ca-nz5rh\" (UID: \"00bf4681-2beb-4489-aba4-7e85f7bf2aaa\") " pod="openshift-image-registry/node-ca-nz5rh"
Aug 04 17:59:49.047331 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:49.045935    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5f4f37d6-a72d-4707-a626-794c55098e25-sys-fs\") pod \"aws-ebs-csi-driver-node-57zmj\" (UID: \"5f4f37d6-a72d-4707-a626-794c55098e25\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-57zmj"
Aug 04 17:59:49.047331 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:49.045962    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/83f26afd-f898-4d64-8d22-4be7a1f5d862-systemd-units\") pod \"ovnkube-node-mzl4m\" (UID: \"83f26afd-f898-4d64-8d22-4be7a1f5d862\") " pod="openshift-ovn-kubernetes/ovnkube-node-mzl4m"
Aug 04 17:59:49.047331 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:49.045986    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkhmf\" (UniqueName: \"kubernetes.io/projected/9fffa81f-5a52-4425-b941-1693ba961fba-kube-api-access-kkhmf\") pod \"node-resolver-ggrqd\" (UID: \"9fffa81f-5a52-4425-b941-1693ba961fba\") " pod="openshift-dns/node-resolver-ggrqd"
Aug 04 17:59:49.047331 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:49.046030    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3145f534-869a-4d4f-b69e-628803f76630-hostroot\") pod \"multus-jwcng\" (UID: \"3145f534-869a-4d4f-b69e-628803f76630\") " pod="openshift-multus/multus-jwcng"
Aug 04 17:59:49.048281 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:49.046058    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/83f26afd-f898-4d64-8d22-4be7a1f5d862-etc-openvswitch\") pod \"ovnkube-node-mzl4m\" (UID: \"83f26afd-f898-4d64-8d22-4be7a1f5d862\") " pod="openshift-ovn-kubernetes/ovnkube-node-mzl4m"
Aug 04 17:59:49.048281 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:49.046077    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/83f26afd-f898-4d64-8d22-4be7a1f5d862-host-cni-netd\") pod \"ovnkube-node-mzl4m\" (UID: \"83f26afd-f898-4d64-8d22-4be7a1f5d862\") " pod="openshift-ovn-kubernetes/ovnkube-node-mzl4m"
Aug 04 17:59:49.048281 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:49.046083    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/31fd88d9-8ce3-4056-a185-4427ec8939c1-original-pull-secret\") pod \"global-pull-secret-syncer-lqxxl\" (UID: \"31fd88d9-8ce3-4056-a185-4427ec8939c1\") " pod="kube-system/global-pull-secret-syncer-lqxxl"
Aug 04 17:59:49.048281 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:49.046127    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/83f26afd-f898-4d64-8d22-4be7a1f5d862-var-lib-openvswitch\") pod \"ovnkube-node-mzl4m\" (UID: \"83f26afd-f898-4d64-8d22-4be7a1f5d862\") " pod="openshift-ovn-kubernetes/ovnkube-node-mzl4m"
Aug 04 17:59:49.048281 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:49.046146    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5f4f37d6-a72d-4707-a626-794c55098e25-sys-fs\") pod \"aws-ebs-csi-driver-node-57zmj\" (UID: \"5f4f37d6-a72d-4707-a626-794c55098e25\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-57zmj"
Aug 04 17:59:49.048281 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:49.046160    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/83f26afd-f898-4d64-8d22-4be7a1f5d862-run-openvswitch\") pod \"ovnkube-node-mzl4m\" (UID: \"83f26afd-f898-4d64-8d22-4be7a1f5d862\") " pod="openshift-ovn-kubernetes/ovnkube-node-mzl4m"
Aug 04 17:59:49.048281 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:49.046187    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/31fd88d9-8ce3-4056-a185-4427ec8939c1-kubelet-config\") pod \"global-pull-secret-syncer-lqxxl\" (UID: \"31fd88d9-8ce3-4056-a185-4427ec8939c1\") " pod="kube-system/global-pull-secret-syncer-lqxxl"
Aug 04 17:59:49.048281 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:49.046192    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/83f26afd-f898-4d64-8d22-4be7a1f5d862-systemd-units\") pod \"ovnkube-node-mzl4m\" (UID: \"83f26afd-f898-4d64-8d22-4be7a1f5d862\") " pod="openshift-ovn-kubernetes/ovnkube-node-mzl4m"
Aug 04 17:59:49.048281 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:49.046213    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3145f534-869a-4d4f-b69e-628803f76630-multus-conf-dir\") pod \"multus-jwcng\" (UID: \"3145f534-869a-4d4f-b69e-628803f76630\") " pod="openshift-multus/multus-jwcng"
Aug 04 17:59:49.048281 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:49.046240    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/83f26afd-f898-4d64-8d22-4be7a1f5d862-host-run-netns\") pod \"ovnkube-node-mzl4m\" (UID: \"83f26afd-f898-4d64-8d22-4be7a1f5d862\") " pod="openshift-ovn-kubernetes/ovnkube-node-mzl4m"
Aug 04 17:59:49.048281 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:49.046262    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9fffa81f-5a52-4425-b941-1693ba961fba-hosts-file\") pod \"node-resolver-ggrqd\" (UID: \"9fffa81f-5a52-4425-b941-1693ba961fba\") " pod="openshift-dns/node-resolver-ggrqd"
Aug 04 17:59:49.048281 ip-10-0-135-124 kubenswrapper[2583]: E0804 17:59:49.046271    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Aug 04 17:59:49.048281 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:49.046276    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3145f534-869a-4d4f-b69e-628803f76630-cni-binary-copy\") pod \"multus-jwcng\" (UID: \"3145f534-869a-4d4f-b69e-628803f76630\") " pod="openshift-multus/multus-jwcng"
Aug 04 17:59:49.048281 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:49.046283    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/83f26afd-f898-4d64-8d22-4be7a1f5d862-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mzl4m\" (UID: \"83f26afd-f898-4d64-8d22-4be7a1f5d862\") " pod="openshift-ovn-kubernetes/ovnkube-node-mzl4m"
Aug 04 17:59:49.048281 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:49.046311    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9fffa81f-5a52-4425-b941-1693ba961fba-tmp-dir\") pod \"node-resolver-ggrqd\" (UID: \"9fffa81f-5a52-4425-b941-1693ba961fba\") " pod="openshift-dns/node-resolver-ggrqd"
Aug 04 17:59:49.048281 ip-10-0-135-124 kubenswrapper[2583]: E0804 17:59:49.046329    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/31fd88d9-8ce3-4056-a185-4427ec8939c1-original-pull-secret podName:31fd88d9-8ce3-4056-a185-4427ec8939c1 nodeName:}" failed. No retries permitted until 2026-08-04 17:59:49.546311485 +0000 UTC m=+3.127530061 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/31fd88d9-8ce3-4056-a185-4427ec8939c1-original-pull-secret") pod "global-pull-secret-syncer-lqxxl" (UID: "31fd88d9-8ce3-4056-a185-4427ec8939c1") : object "kube-system"/"original-pull-secret" not registered
Aug 04 17:59:49.048281 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:49.046352    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3145f534-869a-4d4f-b69e-628803f76630-system-cni-dir\") pod \"multus-jwcng\" (UID: \"3145f534-869a-4d4f-b69e-628803f76630\") " pod="openshift-multus/multus-jwcng"
Aug 04 17:59:49.049190 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:49.046360    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/31fd88d9-8ce3-4056-a185-4427ec8939c1-dbus\") pod \"global-pull-secret-syncer-lqxxl\" (UID: \"31fd88d9-8ce3-4056-a185-4427ec8939c1\") " pod="kube-system/global-pull-secret-syncer-lqxxl"
Aug 04 17:59:49.049190 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:49.046389    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3145f534-869a-4d4f-b69e-628803f76630-multus-cni-dir\") pod \"multus-jwcng\" (UID: \"3145f534-869a-4d4f-b69e-628803f76630\") " pod="openshift-multus/multus-jwcng"
Aug 04 17:59:49.049190 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:49.046415    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/3145f534-869a-4d4f-b69e-628803f76630-os-release\") pod \"multus-jwcng\" (UID: \"3145f534-869a-4d4f-b69e-628803f76630\") " pod="openshift-multus/multus-jwcng"
Aug 04 17:59:49.049190 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:49.046442    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5f4f37d6-a72d-4707-a626-794c55098e25-registration-dir\") pod \"aws-ebs-csi-driver-node-57zmj\" (UID: \"5f4f37d6-a72d-4707-a626-794c55098e25\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-57zmj"
Aug 04 17:59:49.049190 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:49.046488    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/83f26afd-f898-4d64-8d22-4be7a1f5d862-env-overrides\") pod \"ovnkube-node-mzl4m\" (UID: \"83f26afd-f898-4d64-8d22-4be7a1f5d862\") " pod="openshift-ovn-kubernetes/ovnkube-node-mzl4m"
Aug 04 17:59:49.049190 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:49.046517    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/555d14c0-e9e6-4119-8298-a88258e7c516-tuning-conf-dir\") pod \"multus-additional-cni-plugins-c6m8x\" (UID: \"555d14c0-e9e6-4119-8298-a88258e7c516\") " pod="openshift-multus/multus-additional-cni-plugins-c6m8x"
Aug 04 17:59:49.049190 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:49.046565    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/555d14c0-e9e6-4119-8298-a88258e7c516-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-c6m8x\" (UID: \"555d14c0-e9e6-4119-8298-a88258e7c516\") " pod="openshift-multus/multus-additional-cni-plugins-c6m8x"
Aug 04 17:59:49.049190 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:49.046601    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/83f26afd-f898-4d64-8d22-4be7a1f5d862-etc-openvswitch\") pod \"ovnkube-node-mzl4m\" (UID: \"83f26afd-f898-4d64-8d22-4be7a1f5d862\") " pod="openshift-ovn-kubernetes/ovnkube-node-mzl4m"
Aug 04 17:59:49.049190 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:49.046635    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9fffa81f-5a52-4425-b941-1693ba961fba-tmp-dir\") pod \"node-resolver-ggrqd\" (UID: \"9fffa81f-5a52-4425-b941-1693ba961fba\") " pod="openshift-dns/node-resolver-ggrqd"
Aug 04 17:59:49.049190 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:49.046190    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3145f534-869a-4d4f-b69e-628803f76630-multus-socket-dir-parent\") pod \"multus-jwcng\" (UID: \"3145f534-869a-4d4f-b69e-628803f76630\") " pod="openshift-multus/multus-jwcng"
Aug 04 17:59:49.049190 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:49.046662    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3145f534-869a-4d4f-b69e-628803f76630-multus-conf-dir\") pod \"multus-jwcng\" (UID: \"3145f534-869a-4d4f-b69e-628803f76630\") " pod="openshift-multus/multus-jwcng"
Aug 04 17:59:49.049190 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:49.046695    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/83f26afd-f898-4d64-8d22-4be7a1f5d862-var-lib-openvswitch\") pod \"ovnkube-node-mzl4m\" (UID: \"83f26afd-f898-4d64-8d22-4be7a1f5d862\") " pod="openshift-ovn-kubernetes/ovnkube-node-mzl4m"
Aug 04 17:59:49.049190 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:49.046720    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/83f26afd-f898-4d64-8d22-4be7a1f5d862-host-run-netns\") pod \"ovnkube-node-mzl4m\" (UID: \"83f26afd-f898-4d64-8d22-4be7a1f5d862\") " pod="openshift-ovn-kubernetes/ovnkube-node-mzl4m"
Aug 04 17:59:49.049190 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:49.046749    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/83f26afd-f898-4d64-8d22-4be7a1f5d862-run-openvswitch\") pod \"ovnkube-node-mzl4m\" (UID: \"83f26afd-f898-4d64-8d22-4be7a1f5d862\") " pod="openshift-ovn-kubernetes/ovnkube-node-mzl4m"
Aug 04 17:59:49.049190 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:49.046769    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/83f26afd-f898-4d64-8d22-4be7a1f5d862-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mzl4m\" (UID: \"83f26afd-f898-4d64-8d22-4be7a1f5d862\") " pod="openshift-ovn-kubernetes/ovnkube-node-mzl4m"
Aug 04 17:59:49.049190 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:49.046774    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/00bf4681-2beb-4489-aba4-7e85f7bf2aaa-serviceca\") pod \"node-ca-nz5rh\" (UID: \"00bf4681-2beb-4489-aba4-7e85f7bf2aaa\") " pod="openshift-image-registry/node-ca-nz5rh"
Aug 04 17:59:49.049190 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:49.046788    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3145f534-869a-4d4f-b69e-628803f76630-multus-daemon-config\") pod \"multus-jwcng\" (UID: \"3145f534-869a-4d4f-b69e-628803f76630\") " pod="openshift-multus/multus-jwcng"
Aug 04 17:59:49.049985 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:49.046801    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/31fd88d9-8ce3-4056-a185-4427ec8939c1-kubelet-config\") pod \"global-pull-secret-syncer-lqxxl\" (UID: \"31fd88d9-8ce3-4056-a185-4427ec8939c1\") " pod="kube-system/global-pull-secret-syncer-lqxxl"
Aug 04 17:59:49.049985 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:49.046823    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3145f534-869a-4d4f-b69e-628803f76630-hostroot\") pod \"multus-jwcng\" (UID: \"3145f534-869a-4d4f-b69e-628803f76630\") " pod="openshift-multus/multus-jwcng"
Aug 04 17:59:49.049985 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:49.046855    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/3145f534-869a-4d4f-b69e-628803f76630-os-release\") pod \"multus-jwcng\" (UID: \"3145f534-869a-4d4f-b69e-628803f76630\") " pod="openshift-multus/multus-jwcng"
Aug 04 17:59:49.049985 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:49.046861    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5f4f37d6-a72d-4707-a626-794c55098e25-registration-dir\") pod \"aws-ebs-csi-driver-node-57zmj\" (UID: \"5f4f37d6-a72d-4707-a626-794c55098e25\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-57zmj"
Aug 04 17:59:49.049985 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:49.046921    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3145f534-869a-4d4f-b69e-628803f76630-multus-cni-dir\") pod \"multus-jwcng\" (UID: \"3145f534-869a-4d4f-b69e-628803f76630\") " pod="openshift-multus/multus-jwcng"
Aug 04 17:59:49.049985 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:49.046921    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/31fd88d9-8ce3-4056-a185-4427ec8939c1-dbus\") pod \"global-pull-secret-syncer-lqxxl\" (UID: \"31fd88d9-8ce3-4056-a185-4427ec8939c1\") " pod="kube-system/global-pull-secret-syncer-lqxxl"
Aug 04 17:59:49.049985 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:49.047000    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/555d14c0-e9e6-4119-8298-a88258e7c516-tuning-conf-dir\") pod \"multus-additional-cni-plugins-c6m8x\" (UID: \"555d14c0-e9e6-4119-8298-a88258e7c516\") " pod="openshift-multus/multus-additional-cni-plugins-c6m8x"
Aug 04 17:59:49.049985 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:49.047181    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/555d14c0-e9e6-4119-8298-a88258e7c516-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-c6m8x\" (UID: \"555d14c0-e9e6-4119-8298-a88258e7c516\") " pod="openshift-multus/multus-additional-cni-plugins-c6m8x"
Aug 04 17:59:49.049985 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:49.047304    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/83f26afd-f898-4d64-8d22-4be7a1f5d862-env-overrides\") pod \"ovnkube-node-mzl4m\" (UID: \"83f26afd-f898-4d64-8d22-4be7a1f5d862\") " pod="openshift-ovn-kubernetes/ovnkube-node-mzl4m"
Aug 04 17:59:49.049985 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:49.047551    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/83f26afd-f898-4d64-8d22-4be7a1f5d862-ovn-node-metrics-cert\") pod \"ovnkube-node-mzl4m\" (UID: \"83f26afd-f898-4d64-8d22-4be7a1f5d862\") " pod="openshift-ovn-kubernetes/ovnkube-node-mzl4m"
Aug 04 17:59:49.049985 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:49.049685    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/87245716-80f6-43f4-8f06-a0f3924615b2-agent-certs\") pod \"konnectivity-agent-ckl6v\" (UID: \"87245716-80f6-43f4-8f06-a0f3924615b2\") " pod="kube-system/konnectivity-agent-ckl6v"
Aug 04 17:59:49.054228 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:49.054205    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4kzk\" (UniqueName: \"kubernetes.io/projected/5f4f37d6-a72d-4707-a626-794c55098e25-kube-api-access-r4kzk\") pod \"aws-ebs-csi-driver-node-57zmj\" (UID: \"5f4f37d6-a72d-4707-a626-794c55098e25\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-57zmj"
Aug 04 17:59:49.054435 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:49.054415    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4pkwv\" (UniqueName: \"kubernetes.io/projected/0480eb75-3ca1-46ba-9606-ec9e98dc9c4c-kube-api-access-4pkwv\") pod \"network-metrics-daemon-qw2tv\" (UID: \"0480eb75-3ca1-46ba-9606-ec9e98dc9c4c\") " pod="openshift-multus/network-metrics-daemon-qw2tv"
Aug 04 17:59:49.055124 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:49.055079    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xd2jf\" (UniqueName: \"kubernetes.io/projected/3145f534-869a-4d4f-b69e-628803f76630-kube-api-access-xd2jf\") pod \"multus-jwcng\" (UID: \"3145f534-869a-4d4f-b69e-628803f76630\") " pod="openshift-multus/multus-jwcng"
Aug 04 17:59:49.057916 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:49.057892    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkhmf\" (UniqueName: \"kubernetes.io/projected/9fffa81f-5a52-4425-b941-1693ba961fba-kube-api-access-kkhmf\") pod \"node-resolver-ggrqd\" (UID: \"9fffa81f-5a52-4425-b941-1693ba961fba\") " pod="openshift-dns/node-resolver-ggrqd"
Aug 04 17:59:49.061602 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:49.061501    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxd96\" (UniqueName: \"kubernetes.io/projected/00bf4681-2beb-4489-aba4-7e85f7bf2aaa-kube-api-access-fxd96\") pod \"node-ca-nz5rh\" (UID: \"00bf4681-2beb-4489-aba4-7e85f7bf2aaa\") " pod="openshift-image-registry/node-ca-nz5rh"
Aug 04 17:59:49.061770 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:49.061746    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-znbjv\" (UniqueName: \"kubernetes.io/projected/555d14c0-e9e6-4119-8298-a88258e7c516-kube-api-access-znbjv\") pod \"multus-additional-cni-plugins-c6m8x\" (UID: \"555d14c0-e9e6-4119-8298-a88258e7c516\") " pod="openshift-multus/multus-additional-cni-plugins-c6m8x"
Aug 04 17:59:49.062354 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:49.062335    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzdfk\" (UniqueName: \"kubernetes.io/projected/83f26afd-f898-4d64-8d22-4be7a1f5d862-kube-api-access-gzdfk\") pod \"ovnkube-node-mzl4m\" (UID: \"83f26afd-f898-4d64-8d22-4be7a1f5d862\") " pod="openshift-ovn-kubernetes/ovnkube-node-mzl4m"
Aug 04 17:59:49.094536 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:49.094514    2583 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Aug 04 17:59:49.145092 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:49.145063    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-rtcrf"
Aug 04 17:59:49.163711 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:49.163685    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-7bwz5"
Aug 04 17:59:49.165950 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:49.165931    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-mzl4m"
Aug 04 17:59:49.173615 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:49.173595    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-ggrqd"
Aug 04 17:59:49.181459 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:49.181442    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-jwcng"
Aug 04 17:59:49.190340 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:49.190319    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-c6m8x"
Aug 04 17:59:49.199270 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:49.199247    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-nz5rh"
Aug 04 17:59:49.209114 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:49.209091    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-ckl6v"
Aug 04 17:59:49.214982 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:49.214964    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-57zmj"
Aug 04 17:59:49.449764 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:49.449732    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87rpt\" (UniqueName: \"kubernetes.io/projected/43697de6-c2bd-48c6-81eb-1f37fa5d3174-kube-api-access-87rpt\") pod \"network-check-target-jxnpj\" (UID: \"43697de6-c2bd-48c6-81eb-1f37fa5d3174\") " pod="openshift-network-diagnostics/network-check-target-jxnpj"
Aug 04 17:59:49.449946 ip-10-0-135-124 kubenswrapper[2583]: E0804 17:59:49.449871    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Aug 04 17:59:49.449946 ip-10-0-135-124 kubenswrapper[2583]: E0804 17:59:49.449891    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Aug 04 17:59:49.449946 ip-10-0-135-124 kubenswrapper[2583]: E0804 17:59:49.449904    2583 projected.go:196] Error preparing data for projected volume kube-api-access-87rpt for pod openshift-network-diagnostics/network-check-target-jxnpj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 04 17:59:49.450093 ip-10-0-135-124 kubenswrapper[2583]: E0804 17:59:49.449959    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/43697de6-c2bd-48c6-81eb-1f37fa5d3174-kube-api-access-87rpt podName:43697de6-c2bd-48c6-81eb-1f37fa5d3174 nodeName:}" failed. No retries permitted until 2026-08-04 17:59:50.449944275 +0000 UTC m=+4.031162833 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-87rpt" (UniqueName: "kubernetes.io/projected/43697de6-c2bd-48c6-81eb-1f37fa5d3174-kube-api-access-87rpt") pod "network-check-target-jxnpj" (UID: "43697de6-c2bd-48c6-81eb-1f37fa5d3174") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 04 17:59:49.550368 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:49.550324    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0480eb75-3ca1-46ba-9606-ec9e98dc9c4c-metrics-certs\") pod \"network-metrics-daemon-qw2tv\" (UID: \"0480eb75-3ca1-46ba-9606-ec9e98dc9c4c\") " pod="openshift-multus/network-metrics-daemon-qw2tv"
Aug 04 17:59:49.550573 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:49.550417    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/31fd88d9-8ce3-4056-a185-4427ec8939c1-original-pull-secret\") pod \"global-pull-secret-syncer-lqxxl\" (UID: \"31fd88d9-8ce3-4056-a185-4427ec8939c1\") " pod="kube-system/global-pull-secret-syncer-lqxxl"
Aug 04 17:59:49.550573 ip-10-0-135-124 kubenswrapper[2583]: E0804 17:59:49.550520    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 04 17:59:49.550573 ip-10-0-135-124 kubenswrapper[2583]: E0804 17:59:49.550559    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Aug 04 17:59:49.550733 ip-10-0-135-124 kubenswrapper[2583]: E0804 17:59:49.550606    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0480eb75-3ca1-46ba-9606-ec9e98dc9c4c-metrics-certs podName:0480eb75-3ca1-46ba-9606-ec9e98dc9c4c nodeName:}" failed. No retries permitted until 2026-08-04 17:59:50.550584077 +0000 UTC m=+4.131802645 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0480eb75-3ca1-46ba-9606-ec9e98dc9c4c-metrics-certs") pod "network-metrics-daemon-qw2tv" (UID: "0480eb75-3ca1-46ba-9606-ec9e98dc9c4c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 04 17:59:49.550733 ip-10-0-135-124 kubenswrapper[2583]: E0804 17:59:49.550628    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/31fd88d9-8ce3-4056-a185-4427ec8939c1-original-pull-secret podName:31fd88d9-8ce3-4056-a185-4427ec8939c1 nodeName:}" failed. No retries permitted until 2026-08-04 17:59:50.550617926 +0000 UTC m=+4.131836485 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/31fd88d9-8ce3-4056-a185-4427ec8939c1-original-pull-secret") pod "global-pull-secret-syncer-lqxxl" (UID: "31fd88d9-8ce3-4056-a185-4427ec8939c1") : object "kube-system"/"original-pull-secret" not registered
Aug 04 17:59:49.882523 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:49.882417    2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-08-03 17:54:49 +0000 UTC" deadline="2028-04-11 04:54:52.343541848 +0000 UTC"
Aug 04 17:59:49.882523 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:49.882450    2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14770h55m2.4610974s"
Aug 04 17:59:50.094569 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:50.094512    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f4f37d6_a72d_4707_a626_794c55098e25.slice/crio-f9b5dd6686291c8e25e924b13658692aded2bb9c2feb2fda7f4da4661d5a8bfd WatchSource:0}: Error finding container f9b5dd6686291c8e25e924b13658692aded2bb9c2feb2fda7f4da4661d5a8bfd: Status 404 returned error can't find the container with id f9b5dd6686291c8e25e924b13658692aded2bb9c2feb2fda7f4da4661d5a8bfd
Aug 04 17:59:50.097613 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:50.097584    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8dd7a27_922c_4dde_bead_839291637a79.slice/crio-2abf8cac1a895f3425da5e4bbc9955167f031bf2659dea31ccdc2ceca35ed67a WatchSource:0}: Error finding container 2abf8cac1a895f3425da5e4bbc9955167f031bf2659dea31ccdc2ceca35ed67a: Status 404 returned error can't find the container with id 2abf8cac1a895f3425da5e4bbc9955167f031bf2659dea31ccdc2ceca35ed67a
Aug 04 17:59:50.098683 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:50.098657    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3145f534_869a_4d4f_b69e_628803f76630.slice/crio-2d01729c63c5eb3a85b6c8a1f5dfdc67d875f4f7a904bf9ce945662e44605381 WatchSource:0}: Error finding container 2d01729c63c5eb3a85b6c8a1f5dfdc67d875f4f7a904bf9ce945662e44605381: Status 404 returned error can't find the container with id 2d01729c63c5eb3a85b6c8a1f5dfdc67d875f4f7a904bf9ce945662e44605381
Aug 04 17:59:50.101907 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:50.101879    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod555d14c0_e9e6_4119_8298_a88258e7c516.slice/crio-307a8af45b221cad5c7018cf5f87f09e04fb7b262b0d477390ce2a28723f87cd WatchSource:0}: Error finding container 307a8af45b221cad5c7018cf5f87f09e04fb7b262b0d477390ce2a28723f87cd: Status 404 returned error can't find the container with id 307a8af45b221cad5c7018cf5f87f09e04fb7b262b0d477390ce2a28723f87cd
Aug 04 17:59:50.104194 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:50.104166    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fffa81f_5a52_4425_b941_1693ba961fba.slice/crio-03ff04094f031ede42e7efd8b2abe56bf3ad5632198b7538a9c9c57a2b9f9f99 WatchSource:0}: Error finding container 03ff04094f031ede42e7efd8b2abe56bf3ad5632198b7538a9c9c57a2b9f9f99: Status 404 returned error can't find the container with id 03ff04094f031ede42e7efd8b2abe56bf3ad5632198b7538a9c9c57a2b9f9f99
Aug 04 17:59:50.105169 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:50.105142    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde498784_c90b_4e01_bab8_e2ba98d49466.slice/crio-a2cf4d735ccde56f04d0c6cfb7bb004ac56bbae192802b55cbee81c6e0586a32 WatchSource:0}: Error finding container a2cf4d735ccde56f04d0c6cfb7bb004ac56bbae192802b55cbee81c6e0586a32: Status 404 returned error can't find the container with id a2cf4d735ccde56f04d0c6cfb7bb004ac56bbae192802b55cbee81c6e0586a32
Aug 04 17:59:50.107662 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:50.107639    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00bf4681_2beb_4489_aba4_7e85f7bf2aaa.slice/crio-f0802fe58e47a4a8071591b4d02f5b17f79d0b97846a5d6891c1200bad713cee WatchSource:0}: Error finding container f0802fe58e47a4a8071591b4d02f5b17f79d0b97846a5d6891c1200bad713cee: Status 404 returned error can't find the container with id f0802fe58e47a4a8071591b4d02f5b17f79d0b97846a5d6891c1200bad713cee
Aug 04 17:59:50.107917 ip-10-0-135-124 kubenswrapper[2583]: W0804 17:59:50.107892    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87245716_80f6_43f4_8f06_a0f3924615b2.slice/crio-7b5083c6f12512e01e176a987b376d7f5b7aafb85c11e3bc69a47710688b2e16 WatchSource:0}: Error finding container 7b5083c6f12512e01e176a987b376d7f5b7aafb85c11e3bc69a47710688b2e16: Status 404 returned error can't find the container with id 7b5083c6f12512e01e176a987b376d7f5b7aafb85c11e3bc69a47710688b2e16
Aug 04 17:59:50.458946 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:50.458768    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87rpt\" (UniqueName: \"kubernetes.io/projected/43697de6-c2bd-48c6-81eb-1f37fa5d3174-kube-api-access-87rpt\") pod \"network-check-target-jxnpj\" (UID: \"43697de6-c2bd-48c6-81eb-1f37fa5d3174\") " pod="openshift-network-diagnostics/network-check-target-jxnpj"
Aug 04 17:59:50.458946 ip-10-0-135-124 kubenswrapper[2583]: E0804 17:59:50.458935    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Aug 04 17:59:50.459088 ip-10-0-135-124 kubenswrapper[2583]: E0804 17:59:50.458956    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Aug 04 17:59:50.459088 ip-10-0-135-124 kubenswrapper[2583]: E0804 17:59:50.458966    2583 projected.go:196] Error preparing data for projected volume kube-api-access-87rpt for pod openshift-network-diagnostics/network-check-target-jxnpj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 04 17:59:50.459088 ip-10-0-135-124 kubenswrapper[2583]: E0804 17:59:50.459015    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/43697de6-c2bd-48c6-81eb-1f37fa5d3174-kube-api-access-87rpt podName:43697de6-c2bd-48c6-81eb-1f37fa5d3174 nodeName:}" failed. No retries permitted until 2026-08-04 17:59:52.459001064 +0000 UTC m=+6.040219637 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-87rpt" (UniqueName: "kubernetes.io/projected/43697de6-c2bd-48c6-81eb-1f37fa5d3174-kube-api-access-87rpt") pod "network-check-target-jxnpj" (UID: "43697de6-c2bd-48c6-81eb-1f37fa5d3174") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 04 17:59:50.559745 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:50.559691    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/31fd88d9-8ce3-4056-a185-4427ec8939c1-original-pull-secret\") pod \"global-pull-secret-syncer-lqxxl\" (UID: \"31fd88d9-8ce3-4056-a185-4427ec8939c1\") " pod="kube-system/global-pull-secret-syncer-lqxxl"
Aug 04 17:59:50.559919 ip-10-0-135-124 kubenswrapper[2583]: E0804 17:59:50.559848    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Aug 04 17:59:50.559919 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:50.559857    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0480eb75-3ca1-46ba-9606-ec9e98dc9c4c-metrics-certs\") pod \"network-metrics-daemon-qw2tv\" (UID: \"0480eb75-3ca1-46ba-9606-ec9e98dc9c4c\") " pod="openshift-multus/network-metrics-daemon-qw2tv"
Aug 04 17:59:50.559919 ip-10-0-135-124 kubenswrapper[2583]: E0804 17:59:50.559905    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/31fd88d9-8ce3-4056-a185-4427ec8939c1-original-pull-secret podName:31fd88d9-8ce3-4056-a185-4427ec8939c1 nodeName:}" failed. No retries permitted until 2026-08-04 17:59:52.559889081 +0000 UTC m=+6.141107650 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/31fd88d9-8ce3-4056-a185-4427ec8939c1-original-pull-secret") pod "global-pull-secret-syncer-lqxxl" (UID: "31fd88d9-8ce3-4056-a185-4427ec8939c1") : object "kube-system"/"original-pull-secret" not registered
Aug 04 17:59:50.560095 ip-10-0-135-124 kubenswrapper[2583]: E0804 17:59:50.559976    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 04 17:59:50.560095 ip-10-0-135-124 kubenswrapper[2583]: E0804 17:59:50.560027    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0480eb75-3ca1-46ba-9606-ec9e98dc9c4c-metrics-certs podName:0480eb75-3ca1-46ba-9606-ec9e98dc9c4c nodeName:}" failed. No retries permitted until 2026-08-04 17:59:52.560009688 +0000 UTC m=+6.141228248 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0480eb75-3ca1-46ba-9606-ec9e98dc9c4c-metrics-certs") pod "network-metrics-daemon-qw2tv" (UID: "0480eb75-3ca1-46ba-9606-ec9e98dc9c4c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 04 17:59:50.883657 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:50.883563    2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-08-03 17:54:49 +0000 UTC" deadline="2028-02-24 04:42:29.685328251 +0000 UTC"
Aug 04 17:59:50.883657 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:50.883632    2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13642h42m38.801700579s"
Aug 04 17:59:50.940042 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:50.939131    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jxnpj"
Aug 04 17:59:50.940042 ip-10-0-135-124 kubenswrapper[2583]: E0804 17:59:50.939255    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jxnpj" podUID="43697de6-c2bd-48c6-81eb-1f37fa5d3174"
Aug 04 17:59:50.940042 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:50.939686    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lqxxl"
Aug 04 17:59:50.940042 ip-10-0-135-124 kubenswrapper[2583]: E0804 17:59:50.939785    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lqxxl" podUID="31fd88d9-8ce3-4056-a185-4427ec8939c1"
Aug 04 17:59:50.940042 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:50.939861    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qw2tv"
Aug 04 17:59:50.940042 ip-10-0-135-124 kubenswrapper[2583]: E0804 17:59:50.939950    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qw2tv" podUID="0480eb75-3ca1-46ba-9606-ec9e98dc9c4c"
Aug 04 17:59:50.981411 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:50.981335    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-124.ec2.internal" event={"ID":"f727e1c61c596ab926797040f56320d1","Type":"ContainerStarted","Data":"158ba8bb14eac53a77447693630c12aef424bfe1ecf934fc5a561bb3e177cc01"}
Aug 04 17:59:50.994226 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:50.994196    2583 generic.go:356] "Generic (PLEG): container finished" podID="4e7e6ce51e0af982f917261a0b47d870" containerID="5498a7e95178552f91bc5d48d7821f4d7601f109b2cb624e0ccfc102dd239cf0" exitCode=0
Aug 04 17:59:50.994364 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:50.994282    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-124.ec2.internal" event={"ID":"4e7e6ce51e0af982f917261a0b47d870","Type":"ContainerDied","Data":"5498a7e95178552f91bc5d48d7821f4d7601f109b2cb624e0ccfc102dd239cf0"}
Aug 04 17:59:51.004249 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:51.004191    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-124.ec2.internal" podStartSLOduration=3.004171405 podStartE2EDuration="3.004171405s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 17:59:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 17:59:51.003231893 +0000 UTC m=+4.584450477" watchObservedRunningTime="2026-08-04 17:59:51.004171405 +0000 UTC m=+4.585389991"
Aug 04 17:59:51.010337 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:51.010222    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mzl4m" event={"ID":"83f26afd-f898-4d64-8d22-4be7a1f5d862","Type":"ContainerStarted","Data":"32c99dc4f40d809e3e882c7cfe7179dff94383864553274ca27e72c79925d942"}
Aug 04 17:59:51.035207 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:51.035143    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-7bwz5" event={"ID":"de498784-c90b-4e01-bab8-e2ba98d49466","Type":"ContainerStarted","Data":"a2cf4d735ccde56f04d0c6cfb7bb004ac56bbae192802b55cbee81c6e0586a32"}
Aug 04 17:59:51.053429 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:51.053345    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-jwcng" event={"ID":"3145f534-869a-4d4f-b69e-628803f76630","Type":"ContainerStarted","Data":"2d01729c63c5eb3a85b6c8a1f5dfdc67d875f4f7a904bf9ce945662e44605381"}
Aug 04 17:59:51.066306 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:51.066234    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-c6m8x" event={"ID":"555d14c0-e9e6-4119-8298-a88258e7c516","Type":"ContainerStarted","Data":"307a8af45b221cad5c7018cf5f87f09e04fb7b262b0d477390ce2a28723f87cd"}
Aug 04 17:59:51.080029 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:51.079961    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-rtcrf" event={"ID":"a8dd7a27-922c-4dde-bead-839291637a79","Type":"ContainerStarted","Data":"2abf8cac1a895f3425da5e4bbc9955167f031bf2659dea31ccdc2ceca35ed67a"}
Aug 04 17:59:51.094558 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:51.094469    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-57zmj" event={"ID":"5f4f37d6-a72d-4707-a626-794c55098e25","Type":"ContainerStarted","Data":"f9b5dd6686291c8e25e924b13658692aded2bb9c2feb2fda7f4da4661d5a8bfd"}
Aug 04 17:59:51.114794 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:51.114763    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-ckl6v" event={"ID":"87245716-80f6-43f4-8f06-a0f3924615b2","Type":"ContainerStarted","Data":"7b5083c6f12512e01e176a987b376d7f5b7aafb85c11e3bc69a47710688b2e16"}
Aug 04 17:59:51.122239 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:51.122190    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-nz5rh" event={"ID":"00bf4681-2beb-4489-aba4-7e85f7bf2aaa","Type":"ContainerStarted","Data":"f0802fe58e47a4a8071591b4d02f5b17f79d0b97846a5d6891c1200bad713cee"}
Aug 04 17:59:51.140223 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:51.140157    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-ggrqd" event={"ID":"9fffa81f-5a52-4425-b941-1693ba961fba","Type":"ContainerStarted","Data":"03ff04094f031ede42e7efd8b2abe56bf3ad5632198b7538a9c9c57a2b9f9f99"}
Aug 04 17:59:52.166692 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:52.166606    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-124.ec2.internal" event={"ID":"4e7e6ce51e0af982f917261a0b47d870","Type":"ContainerStarted","Data":"d234afb906bf0ab6c378b8800007b374d7d6cff86276b050378bc9fac5725935"}
Aug 04 17:59:52.477546 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:52.477202    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87rpt\" (UniqueName: \"kubernetes.io/projected/43697de6-c2bd-48c6-81eb-1f37fa5d3174-kube-api-access-87rpt\") pod \"network-check-target-jxnpj\" (UID: \"43697de6-c2bd-48c6-81eb-1f37fa5d3174\") " pod="openshift-network-diagnostics/network-check-target-jxnpj"
Aug 04 17:59:52.477546 ip-10-0-135-124 kubenswrapper[2583]: E0804 17:59:52.477458    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Aug 04 17:59:52.477546 ip-10-0-135-124 kubenswrapper[2583]: E0804 17:59:52.477480    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Aug 04 17:59:52.477546 ip-10-0-135-124 kubenswrapper[2583]: E0804 17:59:52.477494    2583 projected.go:196] Error preparing data for projected volume kube-api-access-87rpt for pod openshift-network-diagnostics/network-check-target-jxnpj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 04 17:59:52.478443 ip-10-0-135-124 kubenswrapper[2583]: E0804 17:59:52.478061    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/43697de6-c2bd-48c6-81eb-1f37fa5d3174-kube-api-access-87rpt podName:43697de6-c2bd-48c6-81eb-1f37fa5d3174 nodeName:}" failed. No retries permitted until 2026-08-04 17:59:56.478032392 +0000 UTC m=+10.059250953 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-87rpt" (UniqueName: "kubernetes.io/projected/43697de6-c2bd-48c6-81eb-1f37fa5d3174-kube-api-access-87rpt") pod "network-check-target-jxnpj" (UID: "43697de6-c2bd-48c6-81eb-1f37fa5d3174") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 04 17:59:52.578424 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:52.578387    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0480eb75-3ca1-46ba-9606-ec9e98dc9c4c-metrics-certs\") pod \"network-metrics-daemon-qw2tv\" (UID: \"0480eb75-3ca1-46ba-9606-ec9e98dc9c4c\") " pod="openshift-multus/network-metrics-daemon-qw2tv"
Aug 04 17:59:52.578651 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:52.578476    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/31fd88d9-8ce3-4056-a185-4427ec8939c1-original-pull-secret\") pod \"global-pull-secret-syncer-lqxxl\" (UID: \"31fd88d9-8ce3-4056-a185-4427ec8939c1\") " pod="kube-system/global-pull-secret-syncer-lqxxl"
Aug 04 17:59:52.578651 ip-10-0-135-124 kubenswrapper[2583]: E0804 17:59:52.578616    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Aug 04 17:59:52.578791 ip-10-0-135-124 kubenswrapper[2583]: E0804 17:59:52.578679    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/31fd88d9-8ce3-4056-a185-4427ec8939c1-original-pull-secret podName:31fd88d9-8ce3-4056-a185-4427ec8939c1 nodeName:}" failed. No retries permitted until 2026-08-04 17:59:56.578660266 +0000 UTC m=+10.159878839 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/31fd88d9-8ce3-4056-a185-4427ec8939c1-original-pull-secret") pod "global-pull-secret-syncer-lqxxl" (UID: "31fd88d9-8ce3-4056-a185-4427ec8939c1") : object "kube-system"/"original-pull-secret" not registered
Aug 04 17:59:52.579123 ip-10-0-135-124 kubenswrapper[2583]: E0804 17:59:52.579101    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 04 17:59:52.579193 ip-10-0-135-124 kubenswrapper[2583]: E0804 17:59:52.579165    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0480eb75-3ca1-46ba-9606-ec9e98dc9c4c-metrics-certs podName:0480eb75-3ca1-46ba-9606-ec9e98dc9c4c nodeName:}" failed. No retries permitted until 2026-08-04 17:59:56.579146594 +0000 UTC m=+10.160365170 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0480eb75-3ca1-46ba-9606-ec9e98dc9c4c-metrics-certs") pod "network-metrics-daemon-qw2tv" (UID: "0480eb75-3ca1-46ba-9606-ec9e98dc9c4c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 04 17:59:52.938486 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:52.938439    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jxnpj"
Aug 04 17:59:52.938665 ip-10-0-135-124 kubenswrapper[2583]: E0804 17:59:52.938586    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jxnpj" podUID="43697de6-c2bd-48c6-81eb-1f37fa5d3174"
Aug 04 17:59:52.939038 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:52.938962    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lqxxl"
Aug 04 17:59:52.939138 ip-10-0-135-124 kubenswrapper[2583]: E0804 17:59:52.939047    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lqxxl" podUID="31fd88d9-8ce3-4056-a185-4427ec8939c1"
Aug 04 17:59:52.939138 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:52.939104    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qw2tv"
Aug 04 17:59:52.939246 ip-10-0-135-124 kubenswrapper[2583]: E0804 17:59:52.939182    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qw2tv" podUID="0480eb75-3ca1-46ba-9606-ec9e98dc9c4c"
Aug 04 17:59:54.938873 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:54.938136    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qw2tv"
Aug 04 17:59:54.938873 ip-10-0-135-124 kubenswrapper[2583]: E0804 17:59:54.938279    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qw2tv" podUID="0480eb75-3ca1-46ba-9606-ec9e98dc9c4c"
Aug 04 17:59:54.938873 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:54.938343    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jxnpj"
Aug 04 17:59:54.938873 ip-10-0-135-124 kubenswrapper[2583]: E0804 17:59:54.938412    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jxnpj" podUID="43697de6-c2bd-48c6-81eb-1f37fa5d3174"
Aug 04 17:59:54.938873 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:54.938452    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lqxxl"
Aug 04 17:59:54.938873 ip-10-0-135-124 kubenswrapper[2583]: E0804 17:59:54.938529    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lqxxl" podUID="31fd88d9-8ce3-4056-a185-4427ec8939c1"
Aug 04 17:59:56.516362 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:56.516332    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87rpt\" (UniqueName: \"kubernetes.io/projected/43697de6-c2bd-48c6-81eb-1f37fa5d3174-kube-api-access-87rpt\") pod \"network-check-target-jxnpj\" (UID: \"43697de6-c2bd-48c6-81eb-1f37fa5d3174\") " pod="openshift-network-diagnostics/network-check-target-jxnpj"
Aug 04 17:59:56.516775 ip-10-0-135-124 kubenswrapper[2583]: E0804 17:59:56.516495    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Aug 04 17:59:56.516775 ip-10-0-135-124 kubenswrapper[2583]: E0804 17:59:56.516514    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Aug 04 17:59:56.516775 ip-10-0-135-124 kubenswrapper[2583]: E0804 17:59:56.516527    2583 projected.go:196] Error preparing data for projected volume kube-api-access-87rpt for pod openshift-network-diagnostics/network-check-target-jxnpj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 04 17:59:56.516775 ip-10-0-135-124 kubenswrapper[2583]: E0804 17:59:56.516577    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/43697de6-c2bd-48c6-81eb-1f37fa5d3174-kube-api-access-87rpt podName:43697de6-c2bd-48c6-81eb-1f37fa5d3174 nodeName:}" failed. No retries permitted until 2026-08-04 18:00:04.516564287 +0000 UTC m=+18.097782842 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-87rpt" (UniqueName: "kubernetes.io/projected/43697de6-c2bd-48c6-81eb-1f37fa5d3174-kube-api-access-87rpt") pod "network-check-target-jxnpj" (UID: "43697de6-c2bd-48c6-81eb-1f37fa5d3174") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 04 17:59:56.618148 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:56.617605    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/31fd88d9-8ce3-4056-a185-4427ec8939c1-original-pull-secret\") pod \"global-pull-secret-syncer-lqxxl\" (UID: \"31fd88d9-8ce3-4056-a185-4427ec8939c1\") " pod="kube-system/global-pull-secret-syncer-lqxxl"
Aug 04 17:59:56.618148 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:56.617681    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0480eb75-3ca1-46ba-9606-ec9e98dc9c4c-metrics-certs\") pod \"network-metrics-daemon-qw2tv\" (UID: \"0480eb75-3ca1-46ba-9606-ec9e98dc9c4c\") " pod="openshift-multus/network-metrics-daemon-qw2tv"
Aug 04 17:59:56.618148 ip-10-0-135-124 kubenswrapper[2583]: E0804 17:59:56.617772    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 04 17:59:56.618148 ip-10-0-135-124 kubenswrapper[2583]: E0804 17:59:56.617826    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0480eb75-3ca1-46ba-9606-ec9e98dc9c4c-metrics-certs podName:0480eb75-3ca1-46ba-9606-ec9e98dc9c4c nodeName:}" failed. No retries permitted until 2026-08-04 18:00:04.617813903 +0000 UTC m=+18.199032458 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0480eb75-3ca1-46ba-9606-ec9e98dc9c4c-metrics-certs") pod "network-metrics-daemon-qw2tv" (UID: "0480eb75-3ca1-46ba-9606-ec9e98dc9c4c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 04 17:59:56.618148 ip-10-0-135-124 kubenswrapper[2583]: E0804 17:59:56.617772    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Aug 04 17:59:56.618148 ip-10-0-135-124 kubenswrapper[2583]: E0804 17:59:56.617891    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/31fd88d9-8ce3-4056-a185-4427ec8939c1-original-pull-secret podName:31fd88d9-8ce3-4056-a185-4427ec8939c1 nodeName:}" failed. No retries permitted until 2026-08-04 18:00:04.617878328 +0000 UTC m=+18.199096890 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/31fd88d9-8ce3-4056-a185-4427ec8939c1-original-pull-secret") pod "global-pull-secret-syncer-lqxxl" (UID: "31fd88d9-8ce3-4056-a185-4427ec8939c1") : object "kube-system"/"original-pull-secret" not registered
Aug 04 17:59:56.937431 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:56.937401    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jxnpj"
Aug 04 17:59:56.937649 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:56.937631    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lqxxl"
Aug 04 17:59:56.939112 ip-10-0-135-124 kubenswrapper[2583]: E0804 17:59:56.939080    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lqxxl" podUID="31fd88d9-8ce3-4056-a185-4427ec8939c1"
Aug 04 17:59:56.939112 ip-10-0-135-124 kubenswrapper[2583]: E0804 17:59:56.937684    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jxnpj" podUID="43697de6-c2bd-48c6-81eb-1f37fa5d3174"
Aug 04 17:59:56.939267 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:56.937633    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qw2tv"
Aug 04 17:59:56.939633 ip-10-0-135-124 kubenswrapper[2583]: E0804 17:59:56.939600    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qw2tv" podUID="0480eb75-3ca1-46ba-9606-ec9e98dc9c4c"
Aug 04 17:59:58.935877 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:58.935768    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qw2tv"
Aug 04 17:59:58.935877 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:58.935793    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jxnpj"
Aug 04 17:59:58.936354 ip-10-0-135-124 kubenswrapper[2583]: I0804 17:59:58.935769    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lqxxl"
Aug 04 17:59:58.936841 ip-10-0-135-124 kubenswrapper[2583]: E0804 17:59:58.936508    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qw2tv" podUID="0480eb75-3ca1-46ba-9606-ec9e98dc9c4c"
Aug 04 17:59:58.936841 ip-10-0-135-124 kubenswrapper[2583]: E0804 17:59:58.936741    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jxnpj" podUID="43697de6-c2bd-48c6-81eb-1f37fa5d3174"
Aug 04 17:59:58.936841 ip-10-0-135-124 kubenswrapper[2583]: E0804 17:59:58.936829    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lqxxl" podUID="31fd88d9-8ce3-4056-a185-4427ec8939c1"
Aug 04 18:00:00.935906 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:00.935863    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qw2tv"
Aug 04 18:00:00.935906 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:00.935891    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lqxxl"
Aug 04 18:00:00.936419 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:00.935863    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jxnpj"
Aug 04 18:00:00.936419 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:00:00.936003    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qw2tv" podUID="0480eb75-3ca1-46ba-9606-ec9e98dc9c4c"
Aug 04 18:00:00.936419 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:00:00.936084    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jxnpj" podUID="43697de6-c2bd-48c6-81eb-1f37fa5d3174"
Aug 04 18:00:00.936419 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:00:00.936183    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lqxxl" podUID="31fd88d9-8ce3-4056-a185-4427ec8939c1"
Aug 04 18:00:02.936236 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:02.936205    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lqxxl"
Aug 04 18:00:02.936706 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:02.936205    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jxnpj"
Aug 04 18:00:02.936706 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:02.936330    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qw2tv"
Aug 04 18:00:02.936706 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:00:02.936319    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lqxxl" podUID="31fd88d9-8ce3-4056-a185-4427ec8939c1"
Aug 04 18:00:02.936706 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:00:02.936440    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jxnpj" podUID="43697de6-c2bd-48c6-81eb-1f37fa5d3174"
Aug 04 18:00:02.936706 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:00:02.936601    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qw2tv" podUID="0480eb75-3ca1-46ba-9606-ec9e98dc9c4c"
Aug 04 18:00:04.579119 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:04.579086    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87rpt\" (UniqueName: \"kubernetes.io/projected/43697de6-c2bd-48c6-81eb-1f37fa5d3174-kube-api-access-87rpt\") pod \"network-check-target-jxnpj\" (UID: \"43697de6-c2bd-48c6-81eb-1f37fa5d3174\") " pod="openshift-network-diagnostics/network-check-target-jxnpj"
Aug 04 18:00:04.579545 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:00:04.579205    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Aug 04 18:00:04.579545 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:00:04.579218    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Aug 04 18:00:04.579545 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:00:04.579226    2583 projected.go:196] Error preparing data for projected volume kube-api-access-87rpt for pod openshift-network-diagnostics/network-check-target-jxnpj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 04 18:00:04.579545 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:00:04.579281    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/43697de6-c2bd-48c6-81eb-1f37fa5d3174-kube-api-access-87rpt podName:43697de6-c2bd-48c6-81eb-1f37fa5d3174 nodeName:}" failed. No retries permitted until 2026-08-04 18:00:20.579268403 +0000 UTC m=+34.160486959 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-87rpt" (UniqueName: "kubernetes.io/projected/43697de6-c2bd-48c6-81eb-1f37fa5d3174-kube-api-access-87rpt") pod "network-check-target-jxnpj" (UID: "43697de6-c2bd-48c6-81eb-1f37fa5d3174") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 04 18:00:04.680221 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:04.680195    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0480eb75-3ca1-46ba-9606-ec9e98dc9c4c-metrics-certs\") pod \"network-metrics-daemon-qw2tv\" (UID: \"0480eb75-3ca1-46ba-9606-ec9e98dc9c4c\") " pod="openshift-multus/network-metrics-daemon-qw2tv"
Aug 04 18:00:04.680351 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:04.680239    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/31fd88d9-8ce3-4056-a185-4427ec8939c1-original-pull-secret\") pod \"global-pull-secret-syncer-lqxxl\" (UID: \"31fd88d9-8ce3-4056-a185-4427ec8939c1\") " pod="kube-system/global-pull-secret-syncer-lqxxl"
Aug 04 18:00:04.680407 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:00:04.680351    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 04 18:00:04.680407 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:00:04.680396    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Aug 04 18:00:04.680519 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:00:04.680419    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0480eb75-3ca1-46ba-9606-ec9e98dc9c4c-metrics-certs podName:0480eb75-3ca1-46ba-9606-ec9e98dc9c4c nodeName:}" failed. No retries permitted until 2026-08-04 18:00:20.680399061 +0000 UTC m=+34.261617620 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0480eb75-3ca1-46ba-9606-ec9e98dc9c4c-metrics-certs") pod "network-metrics-daemon-qw2tv" (UID: "0480eb75-3ca1-46ba-9606-ec9e98dc9c4c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 04 18:00:04.680519 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:00:04.680440    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/31fd88d9-8ce3-4056-a185-4427ec8939c1-original-pull-secret podName:31fd88d9-8ce3-4056-a185-4427ec8939c1 nodeName:}" failed. No retries permitted until 2026-08-04 18:00:20.680428663 +0000 UTC m=+34.261647222 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/31fd88d9-8ce3-4056-a185-4427ec8939c1-original-pull-secret") pod "global-pull-secret-syncer-lqxxl" (UID: "31fd88d9-8ce3-4056-a185-4427ec8939c1") : object "kube-system"/"original-pull-secret" not registered
Aug 04 18:00:04.935909 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:04.935886    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jxnpj"
Aug 04 18:00:04.936012 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:04.935887    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qw2tv"
Aug 04 18:00:04.936012 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:00:04.935977    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jxnpj" podUID="43697de6-c2bd-48c6-81eb-1f37fa5d3174"
Aug 04 18:00:04.936084 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:04.935887    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lqxxl"
Aug 04 18:00:04.936084 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:00:04.936062    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qw2tv" podUID="0480eb75-3ca1-46ba-9606-ec9e98dc9c4c"
Aug 04 18:00:04.936144 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:00:04.936129    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lqxxl" podUID="31fd88d9-8ce3-4056-a185-4427ec8939c1"
Aug 04 18:00:06.937228 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:06.937196    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qw2tv"
Aug 04 18:00:06.937718 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:00:06.937315    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qw2tv" podUID="0480eb75-3ca1-46ba-9606-ec9e98dc9c4c"
Aug 04 18:00:06.937718 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:06.937361    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jxnpj"
Aug 04 18:00:06.937718 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:00:06.937490    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jxnpj" podUID="43697de6-c2bd-48c6-81eb-1f37fa5d3174"
Aug 04 18:00:06.937718 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:06.937501    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lqxxl"
Aug 04 18:00:06.937718 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:00:06.937631    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lqxxl" podUID="31fd88d9-8ce3-4056-a185-4427ec8939c1"
Aug 04 18:00:08.936642 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:08.936419    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jxnpj"
Aug 04 18:00:08.937262 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:08.936419    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lqxxl"
Aug 04 18:00:08.937262 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:00:08.936718    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jxnpj" podUID="43697de6-c2bd-48c6-81eb-1f37fa5d3174"
Aug 04 18:00:08.937262 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:00:08.936784    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lqxxl" podUID="31fd88d9-8ce3-4056-a185-4427ec8939c1"
Aug 04 18:00:08.937262 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:08.936518    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qw2tv"
Aug 04 18:00:08.937262 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:00:08.936865    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qw2tv" podUID="0480eb75-3ca1-46ba-9606-ec9e98dc9c4c"
Aug 04 18:00:09.198518 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:09.198494    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-ggrqd" event={"ID":"9fffa81f-5a52-4425-b941-1693ba961fba","Type":"ContainerStarted","Data":"753be121af099ed7591b66033956c988e0690ab4fc988185d06c92457666515d"}
Aug 04 18:00:09.200735 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:09.200715    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mzl4m" event={"ID":"83f26afd-f898-4d64-8d22-4be7a1f5d862","Type":"ContainerStarted","Data":"d5f3a41affa665a429383afee3767de93c8311e9231edb3b4c3846d9c0392222"}
Aug 04 18:00:09.200826 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:09.200786    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mzl4m" event={"ID":"83f26afd-f898-4d64-8d22-4be7a1f5d862","Type":"ContainerStarted","Data":"04569880dc323d1d865297a3e2e6558020afe790e8c3b26764522322b4208df6"}
Aug 04 18:00:09.200869 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:09.200837    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mzl4m" event={"ID":"83f26afd-f898-4d64-8d22-4be7a1f5d862","Type":"ContainerStarted","Data":"0f90eab7ee22abb80d9e416e002c2bd45199d272346dcd3111b9595ece0d7ee2"}
Aug 04 18:00:09.200901 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:09.200884    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mzl4m" event={"ID":"83f26afd-f898-4d64-8d22-4be7a1f5d862","Type":"ContainerStarted","Data":"6477d83ed419376f175a546b494b481a7e2ad8cd01de6fd41139c951c65debd6"}
Aug 04 18:00:09.200937 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:09.200925    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mzl4m" event={"ID":"83f26afd-f898-4d64-8d22-4be7a1f5d862","Type":"ContainerStarted","Data":"75b46c41a154a5fb73dcc9785503888bbaf63eea4fde506b759ce6e2d58e127e"}
Aug 04 18:00:09.201949 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:09.201930    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-7bwz5" event={"ID":"de498784-c90b-4e01-bab8-e2ba98d49466","Type":"ContainerStarted","Data":"c0ef10770c12cb7cba40cc499bf07222b14eaa5aad3dc3096e1a28daddc83cf6"}
Aug 04 18:00:09.203162 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:09.203144    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-jwcng" event={"ID":"3145f534-869a-4d4f-b69e-628803f76630","Type":"ContainerStarted","Data":"df80796da6c3d894a4229aba8c54719319fba0b4e7670d87f6289314a7992fc1"}
Aug 04 18:00:09.204389 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:09.204368    2583 generic.go:356] "Generic (PLEG): container finished" podID="555d14c0-e9e6-4119-8298-a88258e7c516" containerID="e79194d421a8bd7235206ced583edf2dc3af80b93b7bcd80a1b51026ea6e30e3" exitCode=0
Aug 04 18:00:09.204514 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:09.204426    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-c6m8x" event={"ID":"555d14c0-e9e6-4119-8298-a88258e7c516","Type":"ContainerDied","Data":"e79194d421a8bd7235206ced583edf2dc3af80b93b7bcd80a1b51026ea6e30e3"}
Aug 04 18:00:09.205787 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:09.205770    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-57zmj" event={"ID":"5f4f37d6-a72d-4707-a626-794c55098e25","Type":"ContainerStarted","Data":"da30eda7571dbaacf986190c52fffa74a948906774198deb2d7bc08c11981740"}
Aug 04 18:00:09.207134 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:09.207118    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-ckl6v" event={"ID":"87245716-80f6-43f4-8f06-a0f3924615b2","Type":"ContainerStarted","Data":"aaf3308282dd94f7fc9ff5798108d5de09e055a4e153cb79ef1b163dd55cee41"}
Aug 04 18:00:09.208370 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:09.208352    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-nz5rh" event={"ID":"00bf4681-2beb-4489-aba4-7e85f7bf2aaa","Type":"ContainerStarted","Data":"fddc72305e41a3ff38b2de7ab9dfc16948936be0c1516030bc8e3ffbc9dd927c"}
Aug 04 18:00:09.229935 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:09.229898    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-124.ec2.internal" podStartSLOduration=21.229887199 podStartE2EDuration="21.229887199s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 17:59:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 17:59:52.186644015 +0000 UTC m=+5.767862591" watchObservedRunningTime="2026-08-04 18:00:09.229887199 +0000 UTC m=+22.811105777"
Aug 04 18:00:09.249809 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:09.247696    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-7bwz5" podStartSLOduration=3.668947397 podStartE2EDuration="22.247679961s" totalImagesPullingTime="18.578732564s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 17:59:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:00:09.247326046 +0000 UTC m=+22.828544624" watchObservedRunningTime="2026-08-04 18:00:09.247679961 +0000 UTC m=+22.828898543"
Aug 04 18:00:09.263936 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:09.263892    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-nz5rh" podStartSLOduration=3.7068074170000003 podStartE2EDuration="22.263884104s" totalImagesPullingTime="18.557076687s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 17:59:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:00:09.263528083 +0000 UTC m=+22.844746658" watchObservedRunningTime="2026-08-04 18:00:09.263884104 +0000 UTC m=+22.845102739"
Aug 04 18:00:09.293885 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:09.293843    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-ckl6v" podStartSLOduration=7.36410342 podStartE2EDuration="22.29382392s" totalImagesPullingTime="14.9297205s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 17:59:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:00:09.293131556 +0000 UTC m=+22.874350133" watchObservedRunningTime="2026-08-04 18:00:09.29382392 +0000 UTC m=+22.875042498"
Aug 04 18:00:09.372078 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:09.372015    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-jwcng" podStartSLOduration=3.728035534 podStartE2EDuration="22.3720002s" totalImagesPullingTime="18.643964666s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 17:59:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:00:09.371681414 +0000 UTC m=+22.952899992" watchObservedRunningTime="2026-08-04 18:00:09.3720002 +0000 UTC m=+22.953218823"
Aug 04 18:00:09.372435 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:09.372407    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-ggrqd" podStartSLOduration=7.439901793 podStartE2EDuration="22.37239823s" totalImagesPullingTime="14.932496437s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 17:59:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:00:09.351628761 +0000 UTC m=+22.932847338" watchObservedRunningTime="2026-08-04 18:00:09.37239823 +0000 UTC m=+22.953616808"
Aug 04 18:00:10.214323 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:10.214280    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mzl4m" event={"ID":"83f26afd-f898-4d64-8d22-4be7a1f5d862","Type":"ContainerStarted","Data":"59de3acc047734c884222cdb9563ca949d567aa6eb79ff9c3ffca706f4715d23"}
Aug 04 18:00:10.216094 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:10.215993    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-rtcrf" event={"ID":"a8dd7a27-922c-4dde-bead-839291637a79","Type":"ContainerStarted","Data":"7354f95022d7b619a86167a7380f419ee9e2618f2067569d94b836a5855a8f68"}
Aug 04 18:00:10.233417 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:10.233368    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-rtcrf" podStartSLOduration=4.671173642 podStartE2EDuration="23.23334918s" totalImagesPullingTime="18.562175538s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 17:59:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:00:10.232806866 +0000 UTC m=+23.814025444" watchObservedRunningTime="2026-08-04 18:00:10.23334918 +0000 UTC m=+23.814567759"
Aug 04 18:00:10.524365 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:10.524293    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-ckl6v"
Aug 04 18:00:10.524849 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:10.524835    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-ckl6v"
Aug 04 18:00:10.935911 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:10.935882    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lqxxl"
Aug 04 18:00:10.936063 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:00:10.936023    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lqxxl" podUID="31fd88d9-8ce3-4056-a185-4427ec8939c1"
Aug 04 18:00:10.936063 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:10.935882    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qw2tv"
Aug 04 18:00:10.936185 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:00:10.936121    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qw2tv" podUID="0480eb75-3ca1-46ba-9606-ec9e98dc9c4c"
Aug 04 18:00:10.936185 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:10.936143    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jxnpj"
Aug 04 18:00:10.936293 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:00:10.936204    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jxnpj" podUID="43697de6-c2bd-48c6-81eb-1f37fa5d3174"
Aug 04 18:00:11.218336 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:11.218301    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-ckl6v"
Aug 04 18:00:11.218919 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:11.218895    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-ckl6v"
Aug 04 18:00:11.248850 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:11.248830    2583 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Aug 04 18:00:11.908966 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:11.908857    2583 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-08-04T18:00:11.248846485Z","UUID":"3f31fd20-381b-456d-8aa1-9ecf2fee37a8","Handler":null,"Name":"","Endpoint":""}
Aug 04 18:00:11.910747 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:11.910725    2583 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Aug 04 18:00:11.910880 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:11.910761    2583 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Aug 04 18:00:12.222956 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:12.222855    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-57zmj" event={"ID":"5f4f37d6-a72d-4707-a626-794c55098e25","Type":"ContainerStarted","Data":"a9179a4fe1ecb67debe24fe71c9ef7311ec2a4a40d4a373d180ff6285eff055f"}
Aug 04 18:00:12.226782 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:12.226753    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mzl4m" event={"ID":"83f26afd-f898-4d64-8d22-4be7a1f5d862","Type":"ContainerStarted","Data":"0e10edf0d5c826fbf7c9a948189693c4d65d5f073c574e75b54fa67d03cf07c6"}
Aug 04 18:00:12.936117 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:12.935881    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qw2tv"
Aug 04 18:00:12.936392 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:12.935880    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lqxxl"
Aug 04 18:00:12.936392 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:00:12.936228    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qw2tv" podUID="0480eb75-3ca1-46ba-9606-ec9e98dc9c4c"
Aug 04 18:00:12.936392 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:12.935893    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jxnpj"
Aug 04 18:00:12.936392 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:00:12.936298    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lqxxl" podUID="31fd88d9-8ce3-4056-a185-4427ec8939c1"
Aug 04 18:00:12.936392 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:00:12.936364    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jxnpj" podUID="43697de6-c2bd-48c6-81eb-1f37fa5d3174"
Aug 04 18:00:13.232109 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:13.232027    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-57zmj" event={"ID":"5f4f37d6-a72d-4707-a626-794c55098e25","Type":"ContainerStarted","Data":"531d03b0fda3c8d2bd9c25d26df0cc8fbf855aa9d8894a4dbd8c423cdc4a6cfd"}
Aug 04 18:00:13.252174 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:13.252127    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-57zmj" podStartSLOduration=4.411527069 podStartE2EDuration="26.252113613s" totalImagesPullingTime="21.840586544s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 17:59:47 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:00:13.251999421 +0000 UTC m=+26.833217998" watchObservedRunningTime="2026-08-04 18:00:13.252113613 +0000 UTC m=+26.833332191"
Aug 04 18:00:14.936208 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:14.936049    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qw2tv"
Aug 04 18:00:14.936833 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:14.936050    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jxnpj"
Aug 04 18:00:14.936833 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:00:14.936302    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qw2tv" podUID="0480eb75-3ca1-46ba-9606-ec9e98dc9c4c"
Aug 04 18:00:14.936833 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:14.936063    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lqxxl"
Aug 04 18:00:14.936833 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:00:14.936382    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jxnpj" podUID="43697de6-c2bd-48c6-81eb-1f37fa5d3174"
Aug 04 18:00:14.936833 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:00:14.936462    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lqxxl" podUID="31fd88d9-8ce3-4056-a185-4427ec8939c1"
Aug 04 18:00:15.239276 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:15.239202    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mzl4m" event={"ID":"83f26afd-f898-4d64-8d22-4be7a1f5d862","Type":"ContainerStarted","Data":"2ad2f7c19020625f1e97944220dab824d6f27eec6baddfb1d712d0dd3af54c0e"}
Aug 04 18:00:15.239419 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:15.239401    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-mzl4m"
Aug 04 18:00:15.239498 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:15.239459    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-mzl4m"
Aug 04 18:00:15.239556 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:15.239521    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-mzl4m"
Aug 04 18:00:15.241021 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:15.240999    2583 generic.go:356] "Generic (PLEG): container finished" podID="555d14c0-e9e6-4119-8298-a88258e7c516" containerID="ea6f7352c7f11fbc0dc53f08b5c1abc7c4d8967638e66c267dcbe05e9a645fe4" exitCode=0
Aug 04 18:00:15.241131 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:15.241038    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-c6m8x" event={"ID":"555d14c0-e9e6-4119-8298-a88258e7c516","Type":"ContainerDied","Data":"ea6f7352c7f11fbc0dc53f08b5c1abc7c4d8967638e66c267dcbe05e9a645fe4"}
Aug 04 18:00:15.255629 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:15.255608    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mzl4m"
Aug 04 18:00:15.255877 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:15.255861    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mzl4m"
Aug 04 18:00:15.286785 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:15.286747    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-mzl4m" podStartSLOduration=9.684894975 podStartE2EDuration="28.286734922s" totalImagesPullingTime="18.601839947s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 17:59:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:00:15.285299554 +0000 UTC m=+28.866518132" watchObservedRunningTime="2026-08-04 18:00:15.286734922 +0000 UTC m=+28.867953482"
Aug 04 18:00:16.246666 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:16.246641    2583 generic.go:356] "Generic (PLEG): container finished" podID="555d14c0-e9e6-4119-8298-a88258e7c516" containerID="69c200becb7bafd02225c386902745f89eda4321e547f5d33c4bd3ecba1bdc8a" exitCode=0
Aug 04 18:00:16.247062 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:16.246725    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-c6m8x" event={"ID":"555d14c0-e9e6-4119-8298-a88258e7c516","Type":"ContainerDied","Data":"69c200becb7bafd02225c386902745f89eda4321e547f5d33c4bd3ecba1bdc8a"}
Aug 04 18:00:16.348122 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:16.345606    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-lqxxl"]
Aug 04 18:00:16.348122 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:16.345753    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lqxxl"
Aug 04 18:00:16.348122 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:00:16.345873    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lqxxl" podUID="31fd88d9-8ce3-4056-a185-4427ec8939c1"
Aug 04 18:00:16.349722 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:16.349698    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-qw2tv"]
Aug 04 18:00:16.349823 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:16.349796    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qw2tv"
Aug 04 18:00:16.349887 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:00:16.349871    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qw2tv" podUID="0480eb75-3ca1-46ba-9606-ec9e98dc9c4c"
Aug 04 18:00:16.350528 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:16.350511    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-jxnpj"]
Aug 04 18:00:16.350588 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:16.350578    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jxnpj"
Aug 04 18:00:16.350651 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:00:16.350636    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jxnpj" podUID="43697de6-c2bd-48c6-81eb-1f37fa5d3174"
Aug 04 18:00:17.936130 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:17.936101    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qw2tv"
Aug 04 18:00:17.936494 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:17.936139    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jxnpj"
Aug 04 18:00:17.936494 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:17.936148    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lqxxl"
Aug 04 18:00:17.936494 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:00:17.936259    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qw2tv" podUID="0480eb75-3ca1-46ba-9606-ec9e98dc9c4c"
Aug 04 18:00:17.936494 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:00:17.936388    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jxnpj" podUID="43697de6-c2bd-48c6-81eb-1f37fa5d3174"
Aug 04 18:00:17.936677 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:00:17.936516    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lqxxl" podUID="31fd88d9-8ce3-4056-a185-4427ec8939c1"
Aug 04 18:00:18.254057 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:18.253858    2583 generic.go:356] "Generic (PLEG): container finished" podID="555d14c0-e9e6-4119-8298-a88258e7c516" containerID="8c164eedc40e0052d506bd9dc041848596661c11a402b40d875b2b71597f57e7" exitCode=0
Aug 04 18:00:18.254057 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:18.253940    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-c6m8x" event={"ID":"555d14c0-e9e6-4119-8298-a88258e7c516","Type":"ContainerDied","Data":"8c164eedc40e0052d506bd9dc041848596661c11a402b40d875b2b71597f57e7"}
Aug 04 18:00:19.936189 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:19.936158    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jxnpj"
Aug 04 18:00:19.936608 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:19.936165    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qw2tv"
Aug 04 18:00:19.936608 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:00:19.936260    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jxnpj" podUID="43697de6-c2bd-48c6-81eb-1f37fa5d3174"
Aug 04 18:00:19.936608 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:00:19.936372    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qw2tv" podUID="0480eb75-3ca1-46ba-9606-ec9e98dc9c4c"
Aug 04 18:00:19.936608 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:19.936173    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lqxxl"
Aug 04 18:00:19.936608 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:00:19.936463    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lqxxl" podUID="31fd88d9-8ce3-4056-a185-4427ec8939c1"
Aug 04 18:00:20.602064 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:20.601999    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87rpt\" (UniqueName: \"kubernetes.io/projected/43697de6-c2bd-48c6-81eb-1f37fa5d3174-kube-api-access-87rpt\") pod \"network-check-target-jxnpj\" (UID: \"43697de6-c2bd-48c6-81eb-1f37fa5d3174\") " pod="openshift-network-diagnostics/network-check-target-jxnpj"
Aug 04 18:00:20.602229 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:00:20.602144    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Aug 04 18:00:20.602229 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:00:20.602161    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Aug 04 18:00:20.602229 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:00:20.602172    2583 projected.go:196] Error preparing data for projected volume kube-api-access-87rpt for pod openshift-network-diagnostics/network-check-target-jxnpj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 04 18:00:20.602229 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:00:20.602217    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/43697de6-c2bd-48c6-81eb-1f37fa5d3174-kube-api-access-87rpt podName:43697de6-c2bd-48c6-81eb-1f37fa5d3174 nodeName:}" failed. No retries permitted until 2026-08-04 18:00:52.602203025 +0000 UTC m=+66.183421581 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-87rpt" (UniqueName: "kubernetes.io/projected/43697de6-c2bd-48c6-81eb-1f37fa5d3174-kube-api-access-87rpt") pod "network-check-target-jxnpj" (UID: "43697de6-c2bd-48c6-81eb-1f37fa5d3174") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 04 18:00:20.702618 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:20.702564    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/31fd88d9-8ce3-4056-a185-4427ec8939c1-original-pull-secret\") pod \"global-pull-secret-syncer-lqxxl\" (UID: \"31fd88d9-8ce3-4056-a185-4427ec8939c1\") " pod="kube-system/global-pull-secret-syncer-lqxxl"
Aug 04 18:00:20.702823 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:20.702627    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0480eb75-3ca1-46ba-9606-ec9e98dc9c4c-metrics-certs\") pod \"network-metrics-daemon-qw2tv\" (UID: \"0480eb75-3ca1-46ba-9606-ec9e98dc9c4c\") " pod="openshift-multus/network-metrics-daemon-qw2tv"
Aug 04 18:00:20.702823 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:00:20.702721    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Aug 04 18:00:20.702823 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:00:20.702745    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 04 18:00:20.702823 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:00:20.702787    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/31fd88d9-8ce3-4056-a185-4427ec8939c1-original-pull-secret podName:31fd88d9-8ce3-4056-a185-4427ec8939c1 nodeName:}" failed. No retries permitted until 2026-08-04 18:00:52.702769548 +0000 UTC m=+66.283988123 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/31fd88d9-8ce3-4056-a185-4427ec8939c1-original-pull-secret") pod "global-pull-secret-syncer-lqxxl" (UID: "31fd88d9-8ce3-4056-a185-4427ec8939c1") : object "kube-system"/"original-pull-secret" not registered
Aug 04 18:00:20.702823 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:00:20.702809    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0480eb75-3ca1-46ba-9606-ec9e98dc9c4c-metrics-certs podName:0480eb75-3ca1-46ba-9606-ec9e98dc9c4c nodeName:}" failed. No retries permitted until 2026-08-04 18:00:52.702796872 +0000 UTC m=+66.284015429 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0480eb75-3ca1-46ba-9606-ec9e98dc9c4c-metrics-certs") pod "network-metrics-daemon-qw2tv" (UID: "0480eb75-3ca1-46ba-9606-ec9e98dc9c4c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 04 18:00:21.704534 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:21.704503    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-124.ec2.internal" event="NodeReady"
Aug 04 18:00:21.704982 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:21.704652    2583 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Aug 04 18:00:21.740482 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:21.740444    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-74cb995cd-pjjzg"]
Aug 04 18:00:21.745446 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:21.744679    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-6m4nc"]
Aug 04 18:00:21.745446 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:21.744816    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-74cb995cd-pjjzg"
Aug 04 18:00:21.747913 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:21.747888    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Aug 04 18:00:21.748069 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:21.747940    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Aug 04 18:00:21.748069 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:21.747891    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-68qht\""
Aug 04 18:00:21.748191 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:21.748156    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Aug 04 18:00:21.748733 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:21.748712    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-77696685b-7mh27"]
Aug 04 18:00:21.748948 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:21.748931    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-6m4nc"
Aug 04 18:00:21.753358 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:21.753333    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Aug 04 18:00:21.753490 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:21.753460    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Aug 04 18:00:21.754167 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:21.754145    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-85c4d8f7fd-bm9rl"]
Aug 04 18:00:21.754552 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:21.754529    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-77696685b-7mh27"
Aug 04 18:00:21.754878 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:21.754859    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Aug 04 18:00:21.755074 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:21.754863    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-b27vn\""
Aug 04 18:00:21.755074 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:21.754863    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Aug 04 18:00:21.758163 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:21.757398    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Aug 04 18:00:21.758163 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:21.757666    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-v7q5c"]
Aug 04 18:00:21.758163 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:21.757820    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-85c4d8f7fd-bm9rl"
Aug 04 18:00:21.759126 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:21.759101    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Aug 04 18:00:21.760022 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:21.759998    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-lqwwr\""
Aug 04 18:00:21.761020 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:21.760302    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Aug 04 18:00:21.761020 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:21.760577    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Aug 04 18:00:21.761020 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:21.760794    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Aug 04 18:00:21.761171 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:21.761108    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-fzk78"]
Aug 04 18:00:21.761272 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:21.761259    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Aug 04 18:00:21.761624 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:21.761604    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Aug 04 18:00:21.761722 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:21.761654    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Aug 04 18:00:21.761814 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:21.761718    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Aug 04 18:00:21.761814 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:21.761799    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Aug 04 18:00:21.761985 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:21.761967    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-v7q5c"
Aug 04 18:00:21.762902 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:21.762881    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Aug 04 18:00:21.763313 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:21.763296    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-lq4sk\""
Aug 04 18:00:21.764575 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:21.764523    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-784f97bcf6-jwhg9"]
Aug 04 18:00:21.765436 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:21.765326    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-fzk78"
Aug 04 18:00:21.768597 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:21.768320    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-5ccd6b445f-wskfb"]
Aug 04 18:00:21.768597 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:21.768425    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-jwhg9"
Aug 04 18:00:21.771606 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:21.771533    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-tl2x7\""
Aug 04 18:00:21.771848 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:21.771819    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Aug 04 18:00:21.772031 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:21.772014    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Aug 04 18:00:21.772759 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:21.772693    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-6m4nc"]
Aug 04 18:00:21.772759 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:21.772728    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-v7q5c"]
Aug 04 18:00:21.772930 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:21.772803    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-5ccd6b445f-wskfb"
Aug 04 18:00:21.773682 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:21.773665    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-fzk78"]
Aug 04 18:00:21.774054 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:21.774034    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Aug 04 18:00:21.774448 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:21.774426    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-85c4d8f7fd-bm9rl"]
Aug 04 18:00:21.775824 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:21.775808    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-5ccd6b445f-wskfb"]
Aug 04 18:00:21.776244 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:21.776221    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-74cb995cd-pjjzg"]
Aug 04 18:00:21.777114 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:21.777095    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-65bdbbff45-qjxkc"]
Aug 04 18:00:21.781224 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:21.781205    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-qqg4c"]
Aug 04 18:00:21.781337 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:21.781321    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-65bdbbff45-qjxkc"
Aug 04 18:00:21.784954 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:21.784937    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-77696685b-7mh27"]
Aug 04 18:00:21.785044 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:21.785018    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-qqg4c"
Aug 04 18:00:21.785846 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:21.785829    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Aug 04 18:00:21.786502 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:21.786482    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Aug 04 18:00:21.787367 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:21.787347    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Aug 04 18:00:21.788242 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:21.788207    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Aug 04 18:00:21.788556 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:21.788535    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Aug 04 18:00:21.788664 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:21.788588    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Aug 04 18:00:21.788829 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:21.788810    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Aug 04 18:00:21.788936 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:21.788918    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Aug 04 18:00:21.789175 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:21.789159    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-fnlb7\""
Aug 04 18:00:21.789424 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:21.789408    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Aug 04 18:00:21.789554 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:21.789500    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Aug 04 18:00:21.789617 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:21.789584    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-whrgg\""
Aug 04 18:00:21.789680 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:21.789666    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Aug 04 18:00:21.789954 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:21.789928    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Aug 04 18:00:21.790140 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:21.790124    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Aug 04 18:00:21.790232 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:21.790144    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-csqgv\""
Aug 04 18:00:21.793309 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:21.793290    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Aug 04 18:00:21.794215 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:21.794187    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Aug 04 18:00:21.794663 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:21.794511    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-t2p99\""
Aug 04 18:00:21.794857 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:21.794836    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Aug 04 18:00:21.797221 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:21.797195    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Aug 04 18:00:21.799290 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:21.799271    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-7wnlf"]
Aug 04 18:00:21.800023 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:21.799978    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-vbw88\""
Aug 04 18:00:21.804767 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:21.804749    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Aug 04 18:00:21.806019 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:21.806000    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-vlvdv"]
Aug 04 18:00:21.806486 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:21.806449    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-7wnlf"
Aug 04 18:00:21.809673 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:21.809653    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Aug 04 18:00:21.809851 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:21.809832    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Aug 04 18:00:21.810127 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:21.810111    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-snn7x\""
Aug 04 18:00:21.811559 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:21.811540    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-784f97bcf6-jwhg9"]
Aug 04 18:00:21.811657 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:21.811592    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-65bdbbff45-qjxkc"]
Aug 04 18:00:21.811657 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:21.811615    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-qqg4c"]
Aug 04 18:00:21.811657 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:21.811643    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-7wnlf"]
Aug 04 18:00:21.811807 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:21.811663    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-vlvdv"]
Aug 04 18:00:21.811807 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:21.811706    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-vlvdv"
Aug 04 18:00:21.814108 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:21.814092    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Aug 04 18:00:21.814108 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:21.814101    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-lcczt\""
Aug 04 18:00:21.814399 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:21.814383    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Aug 04 18:00:21.814499 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:21.814383    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Aug 04 18:00:21.912738 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:21.912714    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6rm8\" (UniqueName: \"kubernetes.io/projected/7ef28a34-7ec8-49f8-a58c-920661139d9c-kube-api-access-s6rm8\") pod \"cluster-samples-operator-559cbf5745-qqg4c\" (UID: \"7ef28a34-7ec8-49f8-a58c-920661139d9c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-qqg4c"
Aug 04 18:00:21.912858 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:21.912744    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wk4b4\" (UniqueName: \"kubernetes.io/projected/1f245728-8f1a-43fe-9c0a-dde00647c750-kube-api-access-wk4b4\") pod \"network-check-source-65bdbbff45-qjxkc\" (UID: \"1f245728-8f1a-43fe-9c0a-dde00647c750\") " pod="openshift-network-diagnostics/network-check-source-65bdbbff45-qjxkc"
Aug 04 18:00:21.912858 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:21.912770    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tcvjl\" (UniqueName: \"kubernetes.io/projected/e079c95a-25bd-44e2-8f30-44d6cba4add7-kube-api-access-tcvjl\") pod \"cluster-monitoring-operator-6f55c4f58d-fzk78\" (UID: \"e079c95a-25bd-44e2-8f30-44d6cba4add7\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-fzk78"
Aug 04 18:00:21.912858 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:21.912797    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/32978e3c-1e6b-40dc-9dd9-f5bb8f9777ef-serving-cert\") pod \"service-ca-operator-784f97bcf6-jwhg9\" (UID: \"32978e3c-1e6b-40dc-9dd9-f5bb8f9777ef\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-jwhg9"
Aug 04 18:00:21.912858 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:21.912818    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlqf6\" (UniqueName: \"kubernetes.io/projected/88e72b20-e499-46f0-9374-258bf49f39c2-kube-api-access-qlqf6\") pod \"router-default-85c4d8f7fd-bm9rl\" (UID: \"88e72b20-e499-46f0-9374-258bf49f39c2\") " pod="openshift-ingress/router-default-85c4d8f7fd-bm9rl"
Aug 04 18:00:21.912858 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:21.912840    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/88e72b20-e499-46f0-9374-258bf49f39c2-stats-auth\") pod \"router-default-85c4d8f7fd-bm9rl\" (UID: \"88e72b20-e499-46f0-9374-258bf49f39c2\") " pod="openshift-ingress/router-default-85c4d8f7fd-bm9rl"
Aug 04 18:00:21.913380 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:21.912865    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/88e72b20-e499-46f0-9374-258bf49f39c2-service-ca-bundle\") pod \"router-default-85c4d8f7fd-bm9rl\" (UID: \"88e72b20-e499-46f0-9374-258bf49f39c2\") " pod="openshift-ingress/router-default-85c4d8f7fd-bm9rl"
Aug 04 18:00:21.913380 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:21.912886    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/582f0d25-f53a-491b-ae73-d09ed1ab0501-serving-cert\") pod \"kube-storage-version-migrator-operator-7878f4b88c-6m4nc\" (UID: \"582f0d25-f53a-491b-ae73-d09ed1ab0501\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-6m4nc"
Aug 04 18:00:21.913380 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:21.912986    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbs9k\" (UniqueName: \"kubernetes.io/projected/582f0d25-f53a-491b-ae73-d09ed1ab0501-kube-api-access-fbs9k\") pod \"kube-storage-version-migrator-operator-7878f4b88c-6m4nc\" (UID: \"582f0d25-f53a-491b-ae73-d09ed1ab0501\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-6m4nc"
Aug 04 18:00:21.913380 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:21.913076    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/24555f40-5be1-4a70-9e33-ba3c95c1e02f-serving-cert\") pod \"insights-operator-77696685b-7mh27\" (UID: \"24555f40-5be1-4a70-9e33-ba3c95c1e02f\") " pod="openshift-insights/insights-operator-77696685b-7mh27"
Aug 04 18:00:21.913380 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:21.913160    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bg6t6\" (UniqueName: \"kubernetes.io/projected/4704d0e3-b459-422d-92d9-0e9ec9bbe24f-kube-api-access-bg6t6\") pod \"image-registry-74cb995cd-pjjzg\" (UID: \"4704d0e3-b459-422d-92d9-0e9ec9bbe24f\") " pod="openshift-image-registry/image-registry-74cb995cd-pjjzg"
Aug 04 18:00:21.913380 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:21.913187    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6157c2d3-2fd0-4544-88ad-434ac75f8c04-config\") pod \"console-operator-5ccd6b445f-wskfb\" (UID: \"6157c2d3-2fd0-4544-88ad-434ac75f8c04\") " pod="openshift-console-operator/console-operator-5ccd6b445f-wskfb"
Aug 04 18:00:21.913380 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:21.913213    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6157c2d3-2fd0-4544-88ad-434ac75f8c04-serving-cert\") pod \"console-operator-5ccd6b445f-wskfb\" (UID: \"6157c2d3-2fd0-4544-88ad-434ac75f8c04\") " pod="openshift-console-operator/console-operator-5ccd6b445f-wskfb"
Aug 04 18:00:21.913380 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:21.913235    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f46d23da-d73f-4ada-8ac2-1a0e3313e664-cert\") pod \"ingress-canary-vlvdv\" (UID: \"f46d23da-d73f-4ada-8ac2-1a0e3313e664\") " pod="openshift-ingress-canary/ingress-canary-vlvdv"
Aug 04 18:00:21.913380 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:21.913258    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mtk9\" (UniqueName: \"kubernetes.io/projected/f46d23da-d73f-4ada-8ac2-1a0e3313e664-kube-api-access-4mtk9\") pod \"ingress-canary-vlvdv\" (UID: \"f46d23da-d73f-4ada-8ac2-1a0e3313e664\") " pod="openshift-ingress-canary/ingress-canary-vlvdv"
Aug 04 18:00:21.913380 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:21.913298    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4704d0e3-b459-422d-92d9-0e9ec9bbe24f-bound-sa-token\") pod \"image-registry-74cb995cd-pjjzg\" (UID: \"4704d0e3-b459-422d-92d9-0e9ec9bbe24f\") " pod="openshift-image-registry/image-registry-74cb995cd-pjjzg"
Aug 04 18:00:21.913380 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:21.913323    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rfh6\" (UniqueName: \"kubernetes.io/projected/6b6fb1c8-d652-4046-8f8e-4f85375d5d08-kube-api-access-7rfh6\") pod \"volume-data-source-validator-5b8776669-v7q5c\" (UID: \"6b6fb1c8-d652-4046-8f8e-4f85375d5d08\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-v7q5c"
Aug 04 18:00:21.913380 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:21.913348    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6157c2d3-2fd0-4544-88ad-434ac75f8c04-trusted-ca\") pod \"console-operator-5ccd6b445f-wskfb\" (UID: \"6157c2d3-2fd0-4544-88ad-434ac75f8c04\") " pod="openshift-console-operator/console-operator-5ccd6b445f-wskfb"
Aug 04 18:00:21.913380 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:21.913375    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4704d0e3-b459-422d-92d9-0e9ec9bbe24f-installation-pull-secrets\") pod \"image-registry-74cb995cd-pjjzg\" (UID: \"4704d0e3-b459-422d-92d9-0e9ec9bbe24f\") " pod="openshift-image-registry/image-registry-74cb995cd-pjjzg"
Aug 04 18:00:21.914162 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:21.913399    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/24555f40-5be1-4a70-9e33-ba3c95c1e02f-snapshots\") pod \"insights-operator-77696685b-7mh27\" (UID: \"24555f40-5be1-4a70-9e33-ba3c95c1e02f\") " pod="openshift-insights/insights-operator-77696685b-7mh27"
Aug 04 18:00:21.914162 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:21.913424    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/88e72b20-e499-46f0-9374-258bf49f39c2-metrics-certs\") pod \"router-default-85c4d8f7fd-bm9rl\" (UID: \"88e72b20-e499-46f0-9374-258bf49f39c2\") " pod="openshift-ingress/router-default-85c4d8f7fd-bm9rl"
Aug 04 18:00:21.914162 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:21.913447    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ns6j\" (UniqueName: \"kubernetes.io/projected/32978e3c-1e6b-40dc-9dd9-f5bb8f9777ef-kube-api-access-2ns6j\") pod \"service-ca-operator-784f97bcf6-jwhg9\" (UID: \"32978e3c-1e6b-40dc-9dd9-f5bb8f9777ef\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-jwhg9"
Aug 04 18:00:21.914162 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:21.913498    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4704d0e3-b459-422d-92d9-0e9ec9bbe24f-registry-tls\") pod \"image-registry-74cb995cd-pjjzg\" (UID: \"4704d0e3-b459-422d-92d9-0e9ec9bbe24f\") " pod="openshift-image-registry/image-registry-74cb995cd-pjjzg"
Aug 04 18:00:21.914162 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:21.913542    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4704d0e3-b459-422d-92d9-0e9ec9bbe24f-image-registry-private-configuration\") pod \"image-registry-74cb995cd-pjjzg\" (UID: \"4704d0e3-b459-422d-92d9-0e9ec9bbe24f\") " pod="openshift-image-registry/image-registry-74cb995cd-pjjzg"
Aug 04 18:00:21.914162 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:21.913571    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3fb60b38-9cb9-4a07-9334-1b914ddd4a62-metrics-tls\") pod \"dns-default-7wnlf\" (UID: \"3fb60b38-9cb9-4a07-9334-1b914ddd4a62\") " pod="openshift-dns/dns-default-7wnlf"
Aug 04 18:00:21.914162 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:21.913594    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7ef28a34-7ec8-49f8-a58c-920661139d9c-samples-operator-tls\") pod \"cluster-samples-operator-559cbf5745-qqg4c\" (UID: \"7ef28a34-7ec8-49f8-a58c-920661139d9c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-qqg4c"
Aug 04 18:00:21.914162 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:21.913655    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/24555f40-5be1-4a70-9e33-ba3c95c1e02f-service-ca-bundle\") pod \"insights-operator-77696685b-7mh27\" (UID: \"24555f40-5be1-4a70-9e33-ba3c95c1e02f\") " pod="openshift-insights/insights-operator-77696685b-7mh27"
Aug 04 18:00:21.914162 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:21.913682    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/e079c95a-25bd-44e2-8f30-44d6cba4add7-telemetry-config\") pod \"cluster-monitoring-operator-6f55c4f58d-fzk78\" (UID: \"e079c95a-25bd-44e2-8f30-44d6cba4add7\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-fzk78"
Aug 04 18:00:21.914162 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:21.913745    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e079c95a-25bd-44e2-8f30-44d6cba4add7-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6f55c4f58d-fzk78\" (UID: \"e079c95a-25bd-44e2-8f30-44d6cba4add7\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-fzk78"
Aug 04 18:00:21.914162 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:21.913787    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/24555f40-5be1-4a70-9e33-ba3c95c1e02f-tmp\") pod \"insights-operator-77696685b-7mh27\" (UID: \"24555f40-5be1-4a70-9e33-ba3c95c1e02f\") " pod="openshift-insights/insights-operator-77696685b-7mh27"
Aug 04 18:00:21.914162 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:21.913809    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/32978e3c-1e6b-40dc-9dd9-f5bb8f9777ef-config\") pod \"service-ca-operator-784f97bcf6-jwhg9\" (UID: \"32978e3c-1e6b-40dc-9dd9-f5bb8f9777ef\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-jwhg9"
Aug 04 18:00:21.914162 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:21.913833    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4704d0e3-b459-422d-92d9-0e9ec9bbe24f-trusted-ca\") pod \"image-registry-74cb995cd-pjjzg\" (UID: \"4704d0e3-b459-422d-92d9-0e9ec9bbe24f\") " pod="openshift-image-registry/image-registry-74cb995cd-pjjzg"
Aug 04 18:00:21.914162 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:21.913853    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/24555f40-5be1-4a70-9e33-ba3c95c1e02f-trusted-ca-bundle\") pod \"insights-operator-77696685b-7mh27\" (UID: \"24555f40-5be1-4a70-9e33-ba3c95c1e02f\") " pod="openshift-insights/insights-operator-77696685b-7mh27"
Aug 04 18:00:21.914162 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:21.913869    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4704d0e3-b459-422d-92d9-0e9ec9bbe24f-registry-certificates\") pod \"image-registry-74cb995cd-pjjzg\" (UID: \"4704d0e3-b459-422d-92d9-0e9ec9bbe24f\") " pod="openshift-image-registry/image-registry-74cb995cd-pjjzg"
Aug 04 18:00:21.914937 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:21.913891    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/88e72b20-e499-46f0-9374-258bf49f39c2-default-certificate\") pod \"router-default-85c4d8f7fd-bm9rl\" (UID: \"88e72b20-e499-46f0-9374-258bf49f39c2\") " pod="openshift-ingress/router-default-85c4d8f7fd-bm9rl"
Aug 04 18:00:21.914937 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:21.913915    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-llwzw\" (UniqueName: \"kubernetes.io/projected/3fb60b38-9cb9-4a07-9334-1b914ddd4a62-kube-api-access-llwzw\") pod \"dns-default-7wnlf\" (UID: \"3fb60b38-9cb9-4a07-9334-1b914ddd4a62\") " pod="openshift-dns/dns-default-7wnlf"
Aug 04 18:00:21.914937 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:21.913937    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/582f0d25-f53a-491b-ae73-d09ed1ab0501-config\") pod \"kube-storage-version-migrator-operator-7878f4b88c-6m4nc\" (UID: \"582f0d25-f53a-491b-ae73-d09ed1ab0501\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-6m4nc"
Aug 04 18:00:21.914937 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:21.913959    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4704d0e3-b459-422d-92d9-0e9ec9bbe24f-ca-trust-extracted\") pod \"image-registry-74cb995cd-pjjzg\" (UID: \"4704d0e3-b459-422d-92d9-0e9ec9bbe24f\") " pod="openshift-image-registry/image-registry-74cb995cd-pjjzg"
Aug 04 18:00:21.914937 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:21.913985    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fn4bs\" (UniqueName: \"kubernetes.io/projected/24555f40-5be1-4a70-9e33-ba3c95c1e02f-kube-api-access-fn4bs\") pod \"insights-operator-77696685b-7mh27\" (UID: \"24555f40-5be1-4a70-9e33-ba3c95c1e02f\") " pod="openshift-insights/insights-operator-77696685b-7mh27"
Aug 04 18:00:21.914937 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:21.914009    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sgv5w\" (UniqueName: \"kubernetes.io/projected/6157c2d3-2fd0-4544-88ad-434ac75f8c04-kube-api-access-sgv5w\") pod \"console-operator-5ccd6b445f-wskfb\" (UID: \"6157c2d3-2fd0-4544-88ad-434ac75f8c04\") " pod="openshift-console-operator/console-operator-5ccd6b445f-wskfb"
Aug 04 18:00:21.914937 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:21.914035    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3fb60b38-9cb9-4a07-9334-1b914ddd4a62-config-volume\") pod \"dns-default-7wnlf\" (UID: \"3fb60b38-9cb9-4a07-9334-1b914ddd4a62\") " pod="openshift-dns/dns-default-7wnlf"
Aug 04 18:00:21.914937 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:21.914057    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3fb60b38-9cb9-4a07-9334-1b914ddd4a62-tmp-dir\") pod \"dns-default-7wnlf\" (UID: \"3fb60b38-9cb9-4a07-9334-1b914ddd4a62\") " pod="openshift-dns/dns-default-7wnlf"
Aug 04 18:00:21.935934 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:21.935916    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lqxxl"
Aug 04 18:00:21.936034 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:21.935915    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qw2tv"
Aug 04 18:00:21.936110 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:21.935916    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jxnpj"
Aug 04 18:00:21.938835 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:21.938807    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Aug 04 18:00:21.938936 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:21.938850    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Aug 04 18:00:21.938936 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:21.938908    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-qxtnm\""
Aug 04 18:00:21.939060 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:21.938809    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-88fc8\""
Aug 04 18:00:22.015013 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:22.014988    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4704d0e3-b459-422d-92d9-0e9ec9bbe24f-bound-sa-token\") pod \"image-registry-74cb995cd-pjjzg\" (UID: \"4704d0e3-b459-422d-92d9-0e9ec9bbe24f\") " pod="openshift-image-registry/image-registry-74cb995cd-pjjzg"
Aug 04 18:00:22.015139 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:22.015020    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rfh6\" (UniqueName: \"kubernetes.io/projected/6b6fb1c8-d652-4046-8f8e-4f85375d5d08-kube-api-access-7rfh6\") pod \"volume-data-source-validator-5b8776669-v7q5c\" (UID: \"6b6fb1c8-d652-4046-8f8e-4f85375d5d08\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-v7q5c"
Aug 04 18:00:22.015139 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:22.015036    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6157c2d3-2fd0-4544-88ad-434ac75f8c04-trusted-ca\") pod \"console-operator-5ccd6b445f-wskfb\" (UID: \"6157c2d3-2fd0-4544-88ad-434ac75f8c04\") " pod="openshift-console-operator/console-operator-5ccd6b445f-wskfb"
Aug 04 18:00:22.015139 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:22.015055    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4704d0e3-b459-422d-92d9-0e9ec9bbe24f-installation-pull-secrets\") pod \"image-registry-74cb995cd-pjjzg\" (UID: \"4704d0e3-b459-422d-92d9-0e9ec9bbe24f\") " pod="openshift-image-registry/image-registry-74cb995cd-pjjzg"
Aug 04 18:00:22.015139 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:22.015070    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/24555f40-5be1-4a70-9e33-ba3c95c1e02f-snapshots\") pod \"insights-operator-77696685b-7mh27\" (UID: \"24555f40-5be1-4a70-9e33-ba3c95c1e02f\") " pod="openshift-insights/insights-operator-77696685b-7mh27"
Aug 04 18:00:22.015139 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:22.015093    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/88e72b20-e499-46f0-9374-258bf49f39c2-metrics-certs\") pod \"router-default-85c4d8f7fd-bm9rl\" (UID: \"88e72b20-e499-46f0-9374-258bf49f39c2\") " pod="openshift-ingress/router-default-85c4d8f7fd-bm9rl"
Aug 04 18:00:22.015139 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:22.015124    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ns6j\" (UniqueName: \"kubernetes.io/projected/32978e3c-1e6b-40dc-9dd9-f5bb8f9777ef-kube-api-access-2ns6j\") pod \"service-ca-operator-784f97bcf6-jwhg9\" (UID: \"32978e3c-1e6b-40dc-9dd9-f5bb8f9777ef\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-jwhg9"
Aug 04 18:00:22.017044 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:00:22.016317    2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Aug 04 18:00:22.017044 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:00:22.016348    2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-74cb995cd-pjjzg: secret "image-registry-tls" not found
Aug 04 18:00:22.017044 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:00:22.016645    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4704d0e3-b459-422d-92d9-0e9ec9bbe24f-registry-tls podName:4704d0e3-b459-422d-92d9-0e9ec9bbe24f nodeName:}" failed. No retries permitted until 2026-08-04 18:00:22.516617621 +0000 UTC m=+36.097836179 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4704d0e3-b459-422d-92d9-0e9ec9bbe24f-registry-tls") pod "image-registry-74cb995cd-pjjzg" (UID: "4704d0e3-b459-422d-92d9-0e9ec9bbe24f") : secret "image-registry-tls" not found
Aug 04 18:00:22.017044 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:00:22.016893    2583 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Aug 04 18:00:22.017340 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:00:22.017145    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/88e72b20-e499-46f0-9374-258bf49f39c2-metrics-certs podName:88e72b20-e499-46f0-9374-258bf49f39c2 nodeName:}" failed. No retries permitted until 2026-08-04 18:00:22.517114604 +0000 UTC m=+36.098333163 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/88e72b20-e499-46f0-9374-258bf49f39c2-metrics-certs") pod "router-default-85c4d8f7fd-bm9rl" (UID: "88e72b20-e499-46f0-9374-258bf49f39c2") : secret "router-metrics-certs-default" not found
Aug 04 18:00:22.018268 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:22.017436    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/24555f40-5be1-4a70-9e33-ba3c95c1e02f-snapshots\") pod \"insights-operator-77696685b-7mh27\" (UID: \"24555f40-5be1-4a70-9e33-ba3c95c1e02f\") " pod="openshift-insights/insights-operator-77696685b-7mh27"
Aug 04 18:00:22.018268 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:22.015152    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4704d0e3-b459-422d-92d9-0e9ec9bbe24f-registry-tls\") pod \"image-registry-74cb995cd-pjjzg\" (UID: \"4704d0e3-b459-422d-92d9-0e9ec9bbe24f\") " pod="openshift-image-registry/image-registry-74cb995cd-pjjzg"
Aug 04 18:00:22.018268 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:22.017780    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4704d0e3-b459-422d-92d9-0e9ec9bbe24f-image-registry-private-configuration\") pod \"image-registry-74cb995cd-pjjzg\" (UID: \"4704d0e3-b459-422d-92d9-0e9ec9bbe24f\") " pod="openshift-image-registry/image-registry-74cb995cd-pjjzg"
Aug 04 18:00:22.018268 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:22.017832    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3fb60b38-9cb9-4a07-9334-1b914ddd4a62-metrics-tls\") pod \"dns-default-7wnlf\" (UID: \"3fb60b38-9cb9-4a07-9334-1b914ddd4a62\") " pod="openshift-dns/dns-default-7wnlf"
Aug 04 18:00:22.018268 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:22.017859    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7ef28a34-7ec8-49f8-a58c-920661139d9c-samples-operator-tls\") pod \"cluster-samples-operator-559cbf5745-qqg4c\" (UID: \"7ef28a34-7ec8-49f8-a58c-920661139d9c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-qqg4c"
Aug 04 18:00:22.018268 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:22.017905    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/24555f40-5be1-4a70-9e33-ba3c95c1e02f-service-ca-bundle\") pod \"insights-operator-77696685b-7mh27\" (UID: \"24555f40-5be1-4a70-9e33-ba3c95c1e02f\") " pod="openshift-insights/insights-operator-77696685b-7mh27"
Aug 04 18:00:22.018268 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:22.017941    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/e079c95a-25bd-44e2-8f30-44d6cba4add7-telemetry-config\") pod \"cluster-monitoring-operator-6f55c4f58d-fzk78\" (UID: \"e079c95a-25bd-44e2-8f30-44d6cba4add7\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-fzk78"
Aug 04 18:00:22.018268 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:22.017974    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e079c95a-25bd-44e2-8f30-44d6cba4add7-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6f55c4f58d-fzk78\" (UID: \"e079c95a-25bd-44e2-8f30-44d6cba4add7\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-fzk78"
Aug 04 18:00:22.018268 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:22.018019    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/24555f40-5be1-4a70-9e33-ba3c95c1e02f-tmp\") pod \"insights-operator-77696685b-7mh27\" (UID: \"24555f40-5be1-4a70-9e33-ba3c95c1e02f\") " pod="openshift-insights/insights-operator-77696685b-7mh27"
Aug 04 18:00:22.018268 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:22.018051    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/32978e3c-1e6b-40dc-9dd9-f5bb8f9777ef-config\") pod \"service-ca-operator-784f97bcf6-jwhg9\" (UID: \"32978e3c-1e6b-40dc-9dd9-f5bb8f9777ef\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-jwhg9"
Aug 04 18:00:22.018268 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:22.018081    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4704d0e3-b459-422d-92d9-0e9ec9bbe24f-trusted-ca\") pod \"image-registry-74cb995cd-pjjzg\" (UID: \"4704d0e3-b459-422d-92d9-0e9ec9bbe24f\") " pod="openshift-image-registry/image-registry-74cb995cd-pjjzg"
Aug 04 18:00:22.018268 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:22.018109    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/24555f40-5be1-4a70-9e33-ba3c95c1e02f-trusted-ca-bundle\") pod \"insights-operator-77696685b-7mh27\" (UID: \"24555f40-5be1-4a70-9e33-ba3c95c1e02f\") " pod="openshift-insights/insights-operator-77696685b-7mh27"
Aug 04 18:00:22.018268 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:22.018135    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4704d0e3-b459-422d-92d9-0e9ec9bbe24f-registry-certificates\") pod \"image-registry-74cb995cd-pjjzg\" (UID: \"4704d0e3-b459-422d-92d9-0e9ec9bbe24f\") " pod="openshift-image-registry/image-registry-74cb995cd-pjjzg"
Aug 04 18:00:22.018268 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:22.018166    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/88e72b20-e499-46f0-9374-258bf49f39c2-default-certificate\") pod \"router-default-85c4d8f7fd-bm9rl\" (UID: \"88e72b20-e499-46f0-9374-258bf49f39c2\") " pod="openshift-ingress/router-default-85c4d8f7fd-bm9rl"
Aug 04 18:00:22.018268 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:22.018195    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-llwzw\" (UniqueName: \"kubernetes.io/projected/3fb60b38-9cb9-4a07-9334-1b914ddd4a62-kube-api-access-llwzw\") pod \"dns-default-7wnlf\" (UID: \"3fb60b38-9cb9-4a07-9334-1b914ddd4a62\") " pod="openshift-dns/dns-default-7wnlf"
Aug 04 18:00:22.018268 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:22.018226    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/582f0d25-f53a-491b-ae73-d09ed1ab0501-config\") pod \"kube-storage-version-migrator-operator-7878f4b88c-6m4nc\" (UID: \"582f0d25-f53a-491b-ae73-d09ed1ab0501\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-6m4nc"
Aug 04 18:00:22.019063 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:22.018255    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4704d0e3-b459-422d-92d9-0e9ec9bbe24f-ca-trust-extracted\") pod \"image-registry-74cb995cd-pjjzg\" (UID: \"4704d0e3-b459-422d-92d9-0e9ec9bbe24f\") " pod="openshift-image-registry/image-registry-74cb995cd-pjjzg"
Aug 04 18:00:22.019063 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:22.018525    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6157c2d3-2fd0-4544-88ad-434ac75f8c04-trusted-ca\") pod \"console-operator-5ccd6b445f-wskfb\" (UID: \"6157c2d3-2fd0-4544-88ad-434ac75f8c04\") " pod="openshift-console-operator/console-operator-5ccd6b445f-wskfb"
Aug 04 18:00:22.019063 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:22.018543    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/24555f40-5be1-4a70-9e33-ba3c95c1e02f-tmp\") pod \"insights-operator-77696685b-7mh27\" (UID: \"24555f40-5be1-4a70-9e33-ba3c95c1e02f\") " pod="openshift-insights/insights-operator-77696685b-7mh27"
Aug 04 18:00:22.019063 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:22.018593    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fn4bs\" (UniqueName: \"kubernetes.io/projected/24555f40-5be1-4a70-9e33-ba3c95c1e02f-kube-api-access-fn4bs\") pod \"insights-operator-77696685b-7mh27\" (UID: \"24555f40-5be1-4a70-9e33-ba3c95c1e02f\") " pod="openshift-insights/insights-operator-77696685b-7mh27"
Aug 04 18:00:22.019063 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:22.018633    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgv5w\" (UniqueName: \"kubernetes.io/projected/6157c2d3-2fd0-4544-88ad-434ac75f8c04-kube-api-access-sgv5w\") pod \"console-operator-5ccd6b445f-wskfb\" (UID: \"6157c2d3-2fd0-4544-88ad-434ac75f8c04\") " pod="openshift-console-operator/console-operator-5ccd6b445f-wskfb"
Aug 04 18:00:22.019063 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:22.018675    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3fb60b38-9cb9-4a07-9334-1b914ddd4a62-config-volume\") pod \"dns-default-7wnlf\" (UID: \"3fb60b38-9cb9-4a07-9334-1b914ddd4a62\") " pod="openshift-dns/dns-default-7wnlf"
Aug 04 18:00:22.019063 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:22.018708    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3fb60b38-9cb9-4a07-9334-1b914ddd4a62-tmp-dir\") pod \"dns-default-7wnlf\" (UID: \"3fb60b38-9cb9-4a07-9334-1b914ddd4a62\") " pod="openshift-dns/dns-default-7wnlf"
Aug 04 18:00:22.019063 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:22.018763    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6rm8\" (UniqueName: \"kubernetes.io/projected/7ef28a34-7ec8-49f8-a58c-920661139d9c-kube-api-access-s6rm8\") pod \"cluster-samples-operator-559cbf5745-qqg4c\" (UID: \"7ef28a34-7ec8-49f8-a58c-920661139d9c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-qqg4c"
Aug 04 18:00:22.019063 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:22.018802    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wk4b4\" (UniqueName: \"kubernetes.io/projected/1f245728-8f1a-43fe-9c0a-dde00647c750-kube-api-access-wk4b4\") pod \"network-check-source-65bdbbff45-qjxkc\" (UID: \"1f245728-8f1a-43fe-9c0a-dde00647c750\") " pod="openshift-network-diagnostics/network-check-source-65bdbbff45-qjxkc"
Aug 04 18:00:22.019063 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:22.018835    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcvjl\" (UniqueName: \"kubernetes.io/projected/e079c95a-25bd-44e2-8f30-44d6cba4add7-kube-api-access-tcvjl\") pod \"cluster-monitoring-operator-6f55c4f58d-fzk78\" (UID: \"e079c95a-25bd-44e2-8f30-44d6cba4add7\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-fzk78"
Aug 04 18:00:22.019063 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:22.018873    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4704d0e3-b459-422d-92d9-0e9ec9bbe24f-ca-trust-extracted\") pod \"image-registry-74cb995cd-pjjzg\" (UID: \"4704d0e3-b459-422d-92d9-0e9ec9bbe24f\") " pod="openshift-image-registry/image-registry-74cb995cd-pjjzg"
Aug 04 18:00:22.019063 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:00:22.018988    2583 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Aug 04 18:00:22.019063 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:00:22.019055    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e079c95a-25bd-44e2-8f30-44d6cba4add7-cluster-monitoring-operator-tls podName:e079c95a-25bd-44e2-8f30-44d6cba4add7 nodeName:}" failed. No retries permitted until 2026-08-04 18:00:22.519035574 +0000 UTC m=+36.100254143 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/e079c95a-25bd-44e2-8f30-44d6cba4add7-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6f55c4f58d-fzk78" (UID: "e079c95a-25bd-44e2-8f30-44d6cba4add7") : secret "cluster-monitoring-operator-tls" not found
Aug 04 18:00:22.019678 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:22.019140    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4704d0e3-b459-422d-92d9-0e9ec9bbe24f-registry-certificates\") pod \"image-registry-74cb995cd-pjjzg\" (UID: \"4704d0e3-b459-422d-92d9-0e9ec9bbe24f\") " pod="openshift-image-registry/image-registry-74cb995cd-pjjzg"
Aug 04 18:00:22.019728 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:22.019674    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/32978e3c-1e6b-40dc-9dd9-f5bb8f9777ef-config\") pod \"service-ca-operator-784f97bcf6-jwhg9\" (UID: \"32978e3c-1e6b-40dc-9dd9-f5bb8f9777ef\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-jwhg9"
Aug 04 18:00:22.023851 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:22.020033    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3fb60b38-9cb9-4a07-9334-1b914ddd4a62-config-volume\") pod \"dns-default-7wnlf\" (UID: \"3fb60b38-9cb9-4a07-9334-1b914ddd4a62\") " pod="openshift-dns/dns-default-7wnlf"
Aug 04 18:00:22.023851 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:22.020290    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3fb60b38-9cb9-4a07-9334-1b914ddd4a62-tmp-dir\") pod \"dns-default-7wnlf\" (UID: \"3fb60b38-9cb9-4a07-9334-1b914ddd4a62\") " pod="openshift-dns/dns-default-7wnlf"
Aug 04 18:00:22.023851 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:22.020687    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4704d0e3-b459-422d-92d9-0e9ec9bbe24f-trusted-ca\") pod \"image-registry-74cb995cd-pjjzg\" (UID: \"4704d0e3-b459-422d-92d9-0e9ec9bbe24f\") " pod="openshift-image-registry/image-registry-74cb995cd-pjjzg"
Aug 04 18:00:22.023851 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:22.021535    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/24555f40-5be1-4a70-9e33-ba3c95c1e02f-trusted-ca-bundle\") pod \"insights-operator-77696685b-7mh27\" (UID: \"24555f40-5be1-4a70-9e33-ba3c95c1e02f\") " pod="openshift-insights/insights-operator-77696685b-7mh27"
Aug 04 18:00:22.023851 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:22.018873    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/32978e3c-1e6b-40dc-9dd9-f5bb8f9777ef-serving-cert\") pod \"service-ca-operator-784f97bcf6-jwhg9\" (UID: \"32978e3c-1e6b-40dc-9dd9-f5bb8f9777ef\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-jwhg9"
Aug 04 18:00:22.023851 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:22.021558    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/582f0d25-f53a-491b-ae73-d09ed1ab0501-config\") pod \"kube-storage-version-migrator-operator-7878f4b88c-6m4nc\" (UID: \"582f0d25-f53a-491b-ae73-d09ed1ab0501\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-6m4nc"
Aug 04 18:00:22.023851 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:22.021648    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlqf6\" (UniqueName: \"kubernetes.io/projected/88e72b20-e499-46f0-9374-258bf49f39c2-kube-api-access-qlqf6\") pod \"router-default-85c4d8f7fd-bm9rl\" (UID: \"88e72b20-e499-46f0-9374-258bf49f39c2\") " pod="openshift-ingress/router-default-85c4d8f7fd-bm9rl"
Aug 04 18:00:22.023851 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:22.021686    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/88e72b20-e499-46f0-9374-258bf49f39c2-stats-auth\") pod \"router-default-85c4d8f7fd-bm9rl\" (UID: \"88e72b20-e499-46f0-9374-258bf49f39c2\") " pod="openshift-ingress/router-default-85c4d8f7fd-bm9rl"
Aug 04 18:00:22.023851 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:22.021713    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/88e72b20-e499-46f0-9374-258bf49f39c2-service-ca-bundle\") pod \"router-default-85c4d8f7fd-bm9rl\" (UID: \"88e72b20-e499-46f0-9374-258bf49f39c2\") " pod="openshift-ingress/router-default-85c4d8f7fd-bm9rl"
Aug 04 18:00:22.023851 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:22.021735    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/32978e3c-1e6b-40dc-9dd9-f5bb8f9777ef-serving-cert\") pod \"service-ca-operator-784f97bcf6-jwhg9\" (UID: \"32978e3c-1e6b-40dc-9dd9-f5bb8f9777ef\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-jwhg9"
Aug 04 18:00:22.023851 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:22.021749    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/582f0d25-f53a-491b-ae73-d09ed1ab0501-serving-cert\") pod \"kube-storage-version-migrator-operator-7878f4b88c-6m4nc\" (UID: \"582f0d25-f53a-491b-ae73-d09ed1ab0501\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-6m4nc"
Aug 04 18:00:22.023851 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:22.021817    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbs9k\" (UniqueName: \"kubernetes.io/projected/582f0d25-f53a-491b-ae73-d09ed1ab0501-kube-api-access-fbs9k\") pod \"kube-storage-version-migrator-operator-7878f4b88c-6m4nc\" (UID: \"582f0d25-f53a-491b-ae73-d09ed1ab0501\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-6m4nc"
Aug 04 18:00:22.023851 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:22.021860    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/24555f40-5be1-4a70-9e33-ba3c95c1e02f-serving-cert\") pod \"insights-operator-77696685b-7mh27\" (UID: \"24555f40-5be1-4a70-9e33-ba3c95c1e02f\") " pod="openshift-insights/insights-operator-77696685b-7mh27"
Aug 04 18:00:22.023851 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:22.021900    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bg6t6\" (UniqueName: \"kubernetes.io/projected/4704d0e3-b459-422d-92d9-0e9ec9bbe24f-kube-api-access-bg6t6\") pod \"image-registry-74cb995cd-pjjzg\" (UID: \"4704d0e3-b459-422d-92d9-0e9ec9bbe24f\") " pod="openshift-image-registry/image-registry-74cb995cd-pjjzg"
Aug 04 18:00:22.023851 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:22.021934    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6157c2d3-2fd0-4544-88ad-434ac75f8c04-config\") pod \"console-operator-5ccd6b445f-wskfb\" (UID: \"6157c2d3-2fd0-4544-88ad-434ac75f8c04\") " pod="openshift-console-operator/console-operator-5ccd6b445f-wskfb"
Aug 04 18:00:22.023851 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:22.021962    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6157c2d3-2fd0-4544-88ad-434ac75f8c04-serving-cert\") pod \"console-operator-5ccd6b445f-wskfb\" (UID: \"6157c2d3-2fd0-4544-88ad-434ac75f8c04\") " pod="openshift-console-operator/console-operator-5ccd6b445f-wskfb"
Aug 04 18:00:22.024902 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:22.021997    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f46d23da-d73f-4ada-8ac2-1a0e3313e664-cert\") pod \"ingress-canary-vlvdv\" (UID: \"f46d23da-d73f-4ada-8ac2-1a0e3313e664\") " pod="openshift-ingress-canary/ingress-canary-vlvdv"
Aug 04 18:00:22.024902 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:22.022031    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mtk9\" (UniqueName: \"kubernetes.io/projected/f46d23da-d73f-4ada-8ac2-1a0e3313e664-kube-api-access-4mtk9\") pod \"ingress-canary-vlvdv\" (UID: \"f46d23da-d73f-4ada-8ac2-1a0e3313e664\") " pod="openshift-ingress-canary/ingress-canary-vlvdv"
Aug 04 18:00:22.024902 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:00:22.022446    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/88e72b20-e499-46f0-9374-258bf49f39c2-service-ca-bundle podName:88e72b20-e499-46f0-9374-258bf49f39c2 nodeName:}" failed. No retries permitted until 2026-08-04 18:00:22.522418393 +0000 UTC m=+36.103636966 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/88e72b20-e499-46f0-9374-258bf49f39c2-service-ca-bundle") pod "router-default-85c4d8f7fd-bm9rl" (UID: "88e72b20-e499-46f0-9374-258bf49f39c2") : configmap references non-existent config key: service-ca.crt
Aug 04 18:00:22.024902 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:22.023068    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/e079c95a-25bd-44e2-8f30-44d6cba4add7-telemetry-config\") pod \"cluster-monitoring-operator-6f55c4f58d-fzk78\" (UID: \"e079c95a-25bd-44e2-8f30-44d6cba4add7\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-fzk78"
Aug 04 18:00:22.024902 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:22.024399    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4704d0e3-b459-422d-92d9-0e9ec9bbe24f-installation-pull-secrets\") pod \"image-registry-74cb995cd-pjjzg\" (UID: \"4704d0e3-b459-422d-92d9-0e9ec9bbe24f\") " pod="openshift-image-registry/image-registry-74cb995cd-pjjzg"
Aug 04 18:00:22.024902 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:22.024648    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/582f0d25-f53a-491b-ae73-d09ed1ab0501-serving-cert\") pod \"kube-storage-version-migrator-operator-7878f4b88c-6m4nc\" (UID: \"582f0d25-f53a-491b-ae73-d09ed1ab0501\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-6m4nc"
Aug 04 18:00:22.024902 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:22.024737    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/88e72b20-e499-46f0-9374-258bf49f39c2-default-certificate\") pod \"router-default-85c4d8f7fd-bm9rl\" (UID: \"88e72b20-e499-46f0-9374-258bf49f39c2\") " pod="openshift-ingress/router-default-85c4d8f7fd-bm9rl"
Aug 04 18:00:22.024902 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:00:22.024817    2583 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Aug 04 18:00:22.024902 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:00:22.024855    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ef28a34-7ec8-49f8-a58c-920661139d9c-samples-operator-tls podName:7ef28a34-7ec8-49f8-a58c-920661139d9c nodeName:}" failed. No retries permitted until 2026-08-04 18:00:22.524843197 +0000 UTC m=+36.106061753 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/7ef28a34-7ec8-49f8-a58c-920661139d9c-samples-operator-tls") pod "cluster-samples-operator-559cbf5745-qqg4c" (UID: "7ef28a34-7ec8-49f8-a58c-920661139d9c") : secret "samples-operator-tls" not found
Aug 04 18:00:22.025440 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:22.025068    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/88e72b20-e499-46f0-9374-258bf49f39c2-stats-auth\") pod \"router-default-85c4d8f7fd-bm9rl\" (UID: \"88e72b20-e499-46f0-9374-258bf49f39c2\") " pod="openshift-ingress/router-default-85c4d8f7fd-bm9rl"
Aug 04 18:00:22.025664 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:22.025634    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6157c2d3-2fd0-4544-88ad-434ac75f8c04-config\") pod \"console-operator-5ccd6b445f-wskfb\" (UID: \"6157c2d3-2fd0-4544-88ad-434ac75f8c04\") " pod="openshift-console-operator/console-operator-5ccd6b445f-wskfb"
Aug 04 18:00:22.025822 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:00:22.025803    2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Aug 04 18:00:22.026348 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:00:22.025948    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3fb60b38-9cb9-4a07-9334-1b914ddd4a62-metrics-tls podName:3fb60b38-9cb9-4a07-9334-1b914ddd4a62 nodeName:}" failed. No retries permitted until 2026-08-04 18:00:22.525876782 +0000 UTC m=+36.107095358 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3fb60b38-9cb9-4a07-9334-1b914ddd4a62-metrics-tls") pod "dns-default-7wnlf" (UID: "3fb60b38-9cb9-4a07-9334-1b914ddd4a62") : secret "dns-default-metrics-tls" not found
Aug 04 18:00:22.026348 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:22.026122    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4704d0e3-b459-422d-92d9-0e9ec9bbe24f-image-registry-private-configuration\") pod \"image-registry-74cb995cd-pjjzg\" (UID: \"4704d0e3-b459-422d-92d9-0e9ec9bbe24f\") " pod="openshift-image-registry/image-registry-74cb995cd-pjjzg"
Aug 04 18:00:22.027408 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:22.027262    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6157c2d3-2fd0-4544-88ad-434ac75f8c04-serving-cert\") pod \"console-operator-5ccd6b445f-wskfb\" (UID: \"6157c2d3-2fd0-4544-88ad-434ac75f8c04\") " pod="openshift-console-operator/console-operator-5ccd6b445f-wskfb"
Aug 04 18:00:22.028053 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:22.027621    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/24555f40-5be1-4a70-9e33-ba3c95c1e02f-serving-cert\") pod \"insights-operator-77696685b-7mh27\" (UID: \"24555f40-5be1-4a70-9e33-ba3c95c1e02f\") " pod="openshift-insights/insights-operator-77696685b-7mh27"
Aug 04 18:00:22.028053 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:22.027976    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/24555f40-5be1-4a70-9e33-ba3c95c1e02f-service-ca-bundle\") pod \"insights-operator-77696685b-7mh27\" (UID: \"24555f40-5be1-4a70-9e33-ba3c95c1e02f\") " pod="openshift-insights/insights-operator-77696685b-7mh27"
Aug 04 18:00:22.028254 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:22.028224    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgv5w\" (UniqueName: \"kubernetes.io/projected/6157c2d3-2fd0-4544-88ad-434ac75f8c04-kube-api-access-sgv5w\") pod \"console-operator-5ccd6b445f-wskfb\" (UID: \"6157c2d3-2fd0-4544-88ad-434ac75f8c04\") " pod="openshift-console-operator/console-operator-5ccd6b445f-wskfb"
Aug 04 18:00:22.029085 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:22.029061    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fn4bs\" (UniqueName: \"kubernetes.io/projected/24555f40-5be1-4a70-9e33-ba3c95c1e02f-kube-api-access-fn4bs\") pod \"insights-operator-77696685b-7mh27\" (UID: \"24555f40-5be1-4a70-9e33-ba3c95c1e02f\") " pod="openshift-insights/insights-operator-77696685b-7mh27"
Aug 04 18:00:22.029178 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:22.029145    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ns6j\" (UniqueName: \"kubernetes.io/projected/32978e3c-1e6b-40dc-9dd9-f5bb8f9777ef-kube-api-access-2ns6j\") pod \"service-ca-operator-784f97bcf6-jwhg9\" (UID: \"32978e3c-1e6b-40dc-9dd9-f5bb8f9777ef\") " pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-jwhg9"
Aug 04 18:00:22.030278 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:22.030254    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f46d23da-d73f-4ada-8ac2-1a0e3313e664-cert\") pod \"ingress-canary-vlvdv\" (UID: \"f46d23da-d73f-4ada-8ac2-1a0e3313e664\") " pod="openshift-ingress-canary/ingress-canary-vlvdv"
Aug 04 18:00:22.030379 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:22.030308    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4704d0e3-b459-422d-92d9-0e9ec9bbe24f-bound-sa-token\") pod \"image-registry-74cb995cd-pjjzg\" (UID: \"4704d0e3-b459-422d-92d9-0e9ec9bbe24f\") " pod="openshift-image-registry/image-registry-74cb995cd-pjjzg"
Aug 04 18:00:22.031898 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:22.031455    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rfh6\" (UniqueName: \"kubernetes.io/projected/6b6fb1c8-d652-4046-8f8e-4f85375d5d08-kube-api-access-7rfh6\") pod \"volume-data-source-validator-5b8776669-v7q5c\" (UID: \"6b6fb1c8-d652-4046-8f8e-4f85375d5d08\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-v7q5c"
Aug 04 18:00:22.031898 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:22.031818    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tcvjl\" (UniqueName: \"kubernetes.io/projected/e079c95a-25bd-44e2-8f30-44d6cba4add7-kube-api-access-tcvjl\") pod \"cluster-monitoring-operator-6f55c4f58d-fzk78\" (UID: \"e079c95a-25bd-44e2-8f30-44d6cba4add7\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-fzk78"
Aug 04 18:00:22.031898 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:22.031848    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6rm8\" (UniqueName: \"kubernetes.io/projected/7ef28a34-7ec8-49f8-a58c-920661139d9c-kube-api-access-s6rm8\") pod \"cluster-samples-operator-559cbf5745-qqg4c\" (UID: \"7ef28a34-7ec8-49f8-a58c-920661139d9c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-qqg4c"
Aug 04 18:00:22.032938 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:22.032897    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wk4b4\" (UniqueName: \"kubernetes.io/projected/1f245728-8f1a-43fe-9c0a-dde00647c750-kube-api-access-wk4b4\") pod \"network-check-source-65bdbbff45-qjxkc\" (UID: \"1f245728-8f1a-43fe-9c0a-dde00647c750\") " pod="openshift-network-diagnostics/network-check-source-65bdbbff45-qjxkc"
Aug 04 18:00:22.033837 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:22.033794    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-llwzw\" (UniqueName: \"kubernetes.io/projected/3fb60b38-9cb9-4a07-9334-1b914ddd4a62-kube-api-access-llwzw\") pod \"dns-default-7wnlf\" (UID: \"3fb60b38-9cb9-4a07-9334-1b914ddd4a62\") " pod="openshift-dns/dns-default-7wnlf"
Aug 04 18:00:22.034198 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:22.034178    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlqf6\" (UniqueName: \"kubernetes.io/projected/88e72b20-e499-46f0-9374-258bf49f39c2-kube-api-access-qlqf6\") pod \"router-default-85c4d8f7fd-bm9rl\" (UID: \"88e72b20-e499-46f0-9374-258bf49f39c2\") " pod="openshift-ingress/router-default-85c4d8f7fd-bm9rl"
Aug 04 18:00:22.034321 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:22.034278    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbs9k\" (UniqueName: \"kubernetes.io/projected/582f0d25-f53a-491b-ae73-d09ed1ab0501-kube-api-access-fbs9k\") pod \"kube-storage-version-migrator-operator-7878f4b88c-6m4nc\" (UID: \"582f0d25-f53a-491b-ae73-d09ed1ab0501\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-6m4nc"
Aug 04 18:00:22.035495 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:22.035455    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bg6t6\" (UniqueName: \"kubernetes.io/projected/4704d0e3-b459-422d-92d9-0e9ec9bbe24f-kube-api-access-bg6t6\") pod \"image-registry-74cb995cd-pjjzg\" (UID: \"4704d0e3-b459-422d-92d9-0e9ec9bbe24f\") " pod="openshift-image-registry/image-registry-74cb995cd-pjjzg"
Aug 04 18:00:22.035579 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:22.035525    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mtk9\" (UniqueName: \"kubernetes.io/projected/f46d23da-d73f-4ada-8ac2-1a0e3313e664-kube-api-access-4mtk9\") pod \"ingress-canary-vlvdv\" (UID: \"f46d23da-d73f-4ada-8ac2-1a0e3313e664\") " pod="openshift-ingress-canary/ingress-canary-vlvdv"
Aug 04 18:00:22.078601 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:22.078555    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-6m4nc"
Aug 04 18:00:22.087402 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:22.087384    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-77696685b-7mh27"
Aug 04 18:00:22.108741 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:22.108703    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-v7q5c"
Aug 04 18:00:22.125588 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:22.125561    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-jwhg9"
Aug 04 18:00:22.132389 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:22.132177    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-5ccd6b445f-wskfb"
Aug 04 18:00:22.140247 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:22.140225    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-65bdbbff45-qjxkc"
Aug 04 18:00:22.163839 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:22.163803    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-vlvdv"
Aug 04 18:00:22.291195 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:22.288354    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-77696685b-7mh27"]
Aug 04 18:00:22.291195 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:22.290674    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-6m4nc"]
Aug 04 18:00:22.327247 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:22.327178    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-v7q5c"]
Aug 04 18:00:22.358794 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:22.358757    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-65bdbbff45-qjxkc"]
Aug 04 18:00:22.371908 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:22.371884    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-5ccd6b445f-wskfb"]
Aug 04 18:00:22.373771 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:22.373753    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-784f97bcf6-jwhg9"]
Aug 04 18:00:22.529347 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:22.529270    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/88e72b20-e499-46f0-9374-258bf49f39c2-service-ca-bundle\") pod \"router-default-85c4d8f7fd-bm9rl\" (UID: \"88e72b20-e499-46f0-9374-258bf49f39c2\") " pod="openshift-ingress/router-default-85c4d8f7fd-bm9rl"
Aug 04 18:00:22.529347 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:22.529341    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/88e72b20-e499-46f0-9374-258bf49f39c2-metrics-certs\") pod \"router-default-85c4d8f7fd-bm9rl\" (UID: \"88e72b20-e499-46f0-9374-258bf49f39c2\") " pod="openshift-ingress/router-default-85c4d8f7fd-bm9rl"
Aug 04 18:00:22.529581 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:22.529368    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4704d0e3-b459-422d-92d9-0e9ec9bbe24f-registry-tls\") pod \"image-registry-74cb995cd-pjjzg\" (UID: \"4704d0e3-b459-422d-92d9-0e9ec9bbe24f\") " pod="openshift-image-registry/image-registry-74cb995cd-pjjzg"
Aug 04 18:00:22.529581 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:22.529404    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3fb60b38-9cb9-4a07-9334-1b914ddd4a62-metrics-tls\") pod \"dns-default-7wnlf\" (UID: \"3fb60b38-9cb9-4a07-9334-1b914ddd4a62\") " pod="openshift-dns/dns-default-7wnlf"
Aug 04 18:00:22.529581 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:00:22.529459    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/88e72b20-e499-46f0-9374-258bf49f39c2-service-ca-bundle podName:88e72b20-e499-46f0-9374-258bf49f39c2 nodeName:}" failed. No retries permitted until 2026-08-04 18:00:23.52941994 +0000 UTC m=+37.110638522 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/88e72b20-e499-46f0-9374-258bf49f39c2-service-ca-bundle") pod "router-default-85c4d8f7fd-bm9rl" (UID: "88e72b20-e499-46f0-9374-258bf49f39c2") : configmap references non-existent config key: service-ca.crt
Aug 04 18:00:22.529581 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:22.529505    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7ef28a34-7ec8-49f8-a58c-920661139d9c-samples-operator-tls\") pod \"cluster-samples-operator-559cbf5745-qqg4c\" (UID: \"7ef28a34-7ec8-49f8-a58c-920661139d9c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-qqg4c"
Aug 04 18:00:22.529581 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:00:22.529527    2583 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Aug 04 18:00:22.529581 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:22.529564    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e079c95a-25bd-44e2-8f30-44d6cba4add7-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6f55c4f58d-fzk78\" (UID: \"e079c95a-25bd-44e2-8f30-44d6cba4add7\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-fzk78"
Aug 04 18:00:22.529581 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:00:22.529529    2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Aug 04 18:00:22.529581 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:00:22.529577    2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Aug 04 18:00:22.529581 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:00:22.529585    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/88e72b20-e499-46f0-9374-258bf49f39c2-metrics-certs podName:88e72b20-e499-46f0-9374-258bf49f39c2 nodeName:}" failed. No retries permitted until 2026-08-04 18:00:23.529567275 +0000 UTC m=+37.110785843 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/88e72b20-e499-46f0-9374-258bf49f39c2-metrics-certs") pod "router-default-85c4d8f7fd-bm9rl" (UID: "88e72b20-e499-46f0-9374-258bf49f39c2") : secret "router-metrics-certs-default" not found
Aug 04 18:00:22.529916 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:00:22.529589    2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-74cb995cd-pjjzg: secret "image-registry-tls" not found
Aug 04 18:00:22.529916 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:00:22.529632    2583 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Aug 04 18:00:22.529916 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:00:22.529646    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4704d0e3-b459-422d-92d9-0e9ec9bbe24f-registry-tls podName:4704d0e3-b459-422d-92d9-0e9ec9bbe24f nodeName:}" failed. No retries permitted until 2026-08-04 18:00:23.529636527 +0000 UTC m=+37.110855111 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4704d0e3-b459-422d-92d9-0e9ec9bbe24f-registry-tls") pod "image-registry-74cb995cd-pjjzg" (UID: "4704d0e3-b459-422d-92d9-0e9ec9bbe24f") : secret "image-registry-tls" not found
Aug 04 18:00:22.529916 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:00:22.529651    2583 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Aug 04 18:00:22.529916 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:00:22.529713    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3fb60b38-9cb9-4a07-9334-1b914ddd4a62-metrics-tls podName:3fb60b38-9cb9-4a07-9334-1b914ddd4a62 nodeName:}" failed. No retries permitted until 2026-08-04 18:00:23.529694472 +0000 UTC m=+37.110913047 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3fb60b38-9cb9-4a07-9334-1b914ddd4a62-metrics-tls") pod "dns-default-7wnlf" (UID: "3fb60b38-9cb9-4a07-9334-1b914ddd4a62") : secret "dns-default-metrics-tls" not found
Aug 04 18:00:22.529916 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:00:22.529733    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ef28a34-7ec8-49f8-a58c-920661139d9c-samples-operator-tls podName:7ef28a34-7ec8-49f8-a58c-920661139d9c nodeName:}" failed. No retries permitted until 2026-08-04 18:00:23.529722117 +0000 UTC m=+37.110940680 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/7ef28a34-7ec8-49f8-a58c-920661139d9c-samples-operator-tls") pod "cluster-samples-operator-559cbf5745-qqg4c" (UID: "7ef28a34-7ec8-49f8-a58c-920661139d9c") : secret "samples-operator-tls" not found
Aug 04 18:00:22.529916 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:00:22.529749    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e079c95a-25bd-44e2-8f30-44d6cba4add7-cluster-monitoring-operator-tls podName:e079c95a-25bd-44e2-8f30-44d6cba4add7 nodeName:}" failed. No retries permitted until 2026-08-04 18:00:23.529740208 +0000 UTC m=+37.110958771 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/e079c95a-25bd-44e2-8f30-44d6cba4add7-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6f55c4f58d-fzk78" (UID: "e079c95a-25bd-44e2-8f30-44d6cba4add7") : secret "cluster-monitoring-operator-tls" not found
Aug 04 18:00:23.537357 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:23.537317    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/88e72b20-e499-46f0-9374-258bf49f39c2-service-ca-bundle\") pod \"router-default-85c4d8f7fd-bm9rl\" (UID: \"88e72b20-e499-46f0-9374-258bf49f39c2\") " pod="openshift-ingress/router-default-85c4d8f7fd-bm9rl"
Aug 04 18:00:23.538109 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:23.537390    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/88e72b20-e499-46f0-9374-258bf49f39c2-metrics-certs\") pod \"router-default-85c4d8f7fd-bm9rl\" (UID: \"88e72b20-e499-46f0-9374-258bf49f39c2\") " pod="openshift-ingress/router-default-85c4d8f7fd-bm9rl"
Aug 04 18:00:23.538109 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:23.537462    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4704d0e3-b459-422d-92d9-0e9ec9bbe24f-registry-tls\") pod \"image-registry-74cb995cd-pjjzg\" (UID: \"4704d0e3-b459-422d-92d9-0e9ec9bbe24f\") " pod="openshift-image-registry/image-registry-74cb995cd-pjjzg"
Aug 04 18:00:23.538109 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:00:23.537508    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/88e72b20-e499-46f0-9374-258bf49f39c2-service-ca-bundle podName:88e72b20-e499-46f0-9374-258bf49f39c2 nodeName:}" failed. No retries permitted until 2026-08-04 18:00:25.53748498 +0000 UTC m=+39.118703556 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/88e72b20-e499-46f0-9374-258bf49f39c2-service-ca-bundle") pod "router-default-85c4d8f7fd-bm9rl" (UID: "88e72b20-e499-46f0-9374-258bf49f39c2") : configmap references non-existent config key: service-ca.crt
Aug 04 18:00:23.538109 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:23.537558    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3fb60b38-9cb9-4a07-9334-1b914ddd4a62-metrics-tls\") pod \"dns-default-7wnlf\" (UID: \"3fb60b38-9cb9-4a07-9334-1b914ddd4a62\") " pod="openshift-dns/dns-default-7wnlf"
Aug 04 18:00:23.538109 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:00:23.537588    2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Aug 04 18:00:23.538109 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:00:23.537604    2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-74cb995cd-pjjzg: secret "image-registry-tls" not found
Aug 04 18:00:23.538109 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:00:23.537623    2583 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Aug 04 18:00:23.538109 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:00:23.537641    2583 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Aug 04 18:00:23.538109 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:00:23.537652    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4704d0e3-b459-422d-92d9-0e9ec9bbe24f-registry-tls podName:4704d0e3-b459-422d-92d9-0e9ec9bbe24f nodeName:}" failed. No retries permitted until 2026-08-04 18:00:25.537635933 +0000 UTC m=+39.118854522 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4704d0e3-b459-422d-92d9-0e9ec9bbe24f-registry-tls") pod "image-registry-74cb995cd-pjjzg" (UID: "4704d0e3-b459-422d-92d9-0e9ec9bbe24f") : secret "image-registry-tls" not found
Aug 04 18:00:23.538109 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:23.537586    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7ef28a34-7ec8-49f8-a58c-920661139d9c-samples-operator-tls\") pod \"cluster-samples-operator-559cbf5745-qqg4c\" (UID: \"7ef28a34-7ec8-49f8-a58c-920661139d9c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-qqg4c"
Aug 04 18:00:23.538109 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:00:23.537665    2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Aug 04 18:00:23.538109 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:00:23.537669    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/88e72b20-e499-46f0-9374-258bf49f39c2-metrics-certs podName:88e72b20-e499-46f0-9374-258bf49f39c2 nodeName:}" failed. No retries permitted until 2026-08-04 18:00:25.537659899 +0000 UTC m=+39.118878461 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/88e72b20-e499-46f0-9374-258bf49f39c2-metrics-certs") pod "router-default-85c4d8f7fd-bm9rl" (UID: "88e72b20-e499-46f0-9374-258bf49f39c2") : secret "router-metrics-certs-default" not found
Aug 04 18:00:23.538109 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:00:23.537737    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ef28a34-7ec8-49f8-a58c-920661139d9c-samples-operator-tls podName:7ef28a34-7ec8-49f8-a58c-920661139d9c nodeName:}" failed. No retries permitted until 2026-08-04 18:00:25.537723435 +0000 UTC m=+39.118941997 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/7ef28a34-7ec8-49f8-a58c-920661139d9c-samples-operator-tls") pod "cluster-samples-operator-559cbf5745-qqg4c" (UID: "7ef28a34-7ec8-49f8-a58c-920661139d9c") : secret "samples-operator-tls" not found
Aug 04 18:00:23.538109 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:00:23.537755    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3fb60b38-9cb9-4a07-9334-1b914ddd4a62-metrics-tls podName:3fb60b38-9cb9-4a07-9334-1b914ddd4a62 nodeName:}" failed. No retries permitted until 2026-08-04 18:00:25.537746478 +0000 UTC m=+39.118965033 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3fb60b38-9cb9-4a07-9334-1b914ddd4a62-metrics-tls") pod "dns-default-7wnlf" (UID: "3fb60b38-9cb9-4a07-9334-1b914ddd4a62") : secret "dns-default-metrics-tls" not found
Aug 04 18:00:23.538109 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:23.537799    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e079c95a-25bd-44e2-8f30-44d6cba4add7-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6f55c4f58d-fzk78\" (UID: \"e079c95a-25bd-44e2-8f30-44d6cba4add7\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-fzk78"
Aug 04 18:00:23.538109 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:00:23.537951    2583 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Aug 04 18:00:23.538790 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:00:23.537990    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e079c95a-25bd-44e2-8f30-44d6cba4add7-cluster-monitoring-operator-tls podName:e079c95a-25bd-44e2-8f30-44d6cba4add7 nodeName:}" failed. No retries permitted until 2026-08-04 18:00:25.537977232 +0000 UTC m=+39.119195788 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/e079c95a-25bd-44e2-8f30-44d6cba4add7-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6f55c4f58d-fzk78" (UID: "e079c95a-25bd-44e2-8f30-44d6cba4add7") : secret "cluster-monitoring-operator-tls" not found
Aug 04 18:00:23.776320 ip-10-0-135-124 kubenswrapper[2583]: W0804 18:00:23.776285    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24555f40_5be1_4a70_9e33_ba3c95c1e02f.slice/crio-b116fffedf327affe833a9275c49ac098cd9e67e409d40ff588aa1dd3e981847 WatchSource:0}: Error finding container b116fffedf327affe833a9275c49ac098cd9e67e409d40ff588aa1dd3e981847: Status 404 returned error can't find the container with id b116fffedf327affe833a9275c49ac098cd9e67e409d40ff588aa1dd3e981847
Aug 04 18:00:23.777595 ip-10-0-135-124 kubenswrapper[2583]: W0804 18:00:23.777264    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32978e3c_1e6b_40dc_9dd9_f5bb8f9777ef.slice/crio-1873f503fd172505479210ffe1e8f1c24a23cb7878c3af39437934613216b9d5 WatchSource:0}: Error finding container 1873f503fd172505479210ffe1e8f1c24a23cb7878c3af39437934613216b9d5: Status 404 returned error can't find the container with id 1873f503fd172505479210ffe1e8f1c24a23cb7878c3af39437934613216b9d5
Aug 04 18:00:23.778227 ip-10-0-135-124 kubenswrapper[2583]: W0804 18:00:23.778182    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b6fb1c8_d652_4046_8f8e_4f85375d5d08.slice/crio-9e9c101749db1923a3b575d023efa3cb1b23a2abe63062cb20a57c1c9c5b7b48 WatchSource:0}: Error finding container 9e9c101749db1923a3b575d023efa3cb1b23a2abe63062cb20a57c1c9c5b7b48: Status 404 returned error can't find the container with id 9e9c101749db1923a3b575d023efa3cb1b23a2abe63062cb20a57c1c9c5b7b48
Aug 04 18:00:23.779071 ip-10-0-135-124 kubenswrapper[2583]: W0804 18:00:23.779033    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f245728_8f1a_43fe_9c0a_dde00647c750.slice/crio-3b14a9a4162ce9f6c114b5cc62e2e82546857a1032b14a31e2b04711a5fb3d68 WatchSource:0}: Error finding container 3b14a9a4162ce9f6c114b5cc62e2e82546857a1032b14a31e2b04711a5fb3d68: Status 404 returned error can't find the container with id 3b14a9a4162ce9f6c114b5cc62e2e82546857a1032b14a31e2b04711a5fb3d68
Aug 04 18:00:23.781522 ip-10-0-135-124 kubenswrapper[2583]: W0804 18:00:23.780948    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod582f0d25_f53a_491b_ae73_d09ed1ab0501.slice/crio-cf8f5ffce9b2ad5319516699aa6014ff53ab0697bfb4420023e3c6f0dfb3cdcb WatchSource:0}: Error finding container cf8f5ffce9b2ad5319516699aa6014ff53ab0697bfb4420023e3c6f0dfb3cdcb: Status 404 returned error can't find the container with id cf8f5ffce9b2ad5319516699aa6014ff53ab0697bfb4420023e3c6f0dfb3cdcb
Aug 04 18:00:23.782065 ip-10-0-135-124 kubenswrapper[2583]: W0804 18:00:23.781238    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6157c2d3_2fd0_4544_88ad_434ac75f8c04.slice/crio-51d97637446783d30a50b1abb00fb739531293c6779d92cfa90fb4c0a3d90b84 WatchSource:0}: Error finding container 51d97637446783d30a50b1abb00fb739531293c6779d92cfa90fb4c0a3d90b84: Status 404 returned error can't find the container with id 51d97637446783d30a50b1abb00fb739531293c6779d92cfa90fb4c0a3d90b84
Aug 04 18:00:23.910164 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:23.910121    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-vlvdv"]
Aug 04 18:00:23.913161 ip-10-0-135-124 kubenswrapper[2583]: W0804 18:00:23.913138    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf46d23da_d73f_4ada_8ac2_1a0e3313e664.slice/crio-fc8dde314bfaf578ecdacd092fae730fb53e7aadc1e189d161624b072e478082 WatchSource:0}: Error finding container fc8dde314bfaf578ecdacd092fae730fb53e7aadc1e189d161624b072e478082: Status 404 returned error can't find the container with id fc8dde314bfaf578ecdacd092fae730fb53e7aadc1e189d161624b072e478082
Aug 04 18:00:24.122598 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:24.122541    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-f77f48c6d-lt6n7"]
Aug 04 18:00:24.125552 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:24.125535    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-f77f48c6d-lt6n7"
Aug 04 18:00:24.128424 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:24.128402    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-865jq\""
Aug 04 18:00:24.128544 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:24.128516    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Aug 04 18:00:24.129623 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:24.129605    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Aug 04 18:00:24.134127 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:24.134106    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-f77f48c6d-lt6n7"]
Aug 04 18:00:24.244912 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:24.244879    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/6d05efcb-b6fb-485c-b472-bd406b8e729f-nginx-conf\") pod \"networking-console-plugin-f77f48c6d-lt6n7\" (UID: \"6d05efcb-b6fb-485c-b472-bd406b8e729f\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-lt6n7"
Aug 04 18:00:24.245122 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:24.244934    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6d05efcb-b6fb-485c-b472-bd406b8e729f-networking-console-plugin-cert\") pod \"networking-console-plugin-f77f48c6d-lt6n7\" (UID: \"6d05efcb-b6fb-485c-b472-bd406b8e729f\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-lt6n7"
Aug 04 18:00:24.270270 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:24.270236    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-jwhg9" event={"ID":"32978e3c-1e6b-40dc-9dd9-f5bb8f9777ef","Type":"ContainerStarted","Data":"1873f503fd172505479210ffe1e8f1c24a23cb7878c3af39437934613216b9d5"}
Aug 04 18:00:24.271835 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:24.271782    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-v7q5c" event={"ID":"6b6fb1c8-d652-4046-8f8e-4f85375d5d08","Type":"ContainerStarted","Data":"9e9c101749db1923a3b575d023efa3cb1b23a2abe63062cb20a57c1c9c5b7b48"}
Aug 04 18:00:24.275612 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:24.275583    2583 generic.go:356] "Generic (PLEG): container finished" podID="555d14c0-e9e6-4119-8298-a88258e7c516" containerID="6666c7ad6cf9c99e72277685a6eaf74e2e2daadb60842fa1d6f7280d78f35cb9" exitCode=0
Aug 04 18:00:24.275713 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:24.275671    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-c6m8x" event={"ID":"555d14c0-e9e6-4119-8298-a88258e7c516","Type":"ContainerDied","Data":"6666c7ad6cf9c99e72277685a6eaf74e2e2daadb60842fa1d6f7280d78f35cb9"}
Aug 04 18:00:24.277186 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:24.277126    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-vlvdv" event={"ID":"f46d23da-d73f-4ada-8ac2-1a0e3313e664","Type":"ContainerStarted","Data":"fc8dde314bfaf578ecdacd092fae730fb53e7aadc1e189d161624b072e478082"}
Aug 04 18:00:24.279119 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:24.279094    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-6m4nc" event={"ID":"582f0d25-f53a-491b-ae73-d09ed1ab0501","Type":"ContainerStarted","Data":"cf8f5ffce9b2ad5319516699aa6014ff53ab0697bfb4420023e3c6f0dfb3cdcb"}
Aug 04 18:00:24.282534 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:24.282461    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-5ccd6b445f-wskfb" event={"ID":"6157c2d3-2fd0-4544-88ad-434ac75f8c04","Type":"ContainerStarted","Data":"51d97637446783d30a50b1abb00fb739531293c6779d92cfa90fb4c0a3d90b84"}
Aug 04 18:00:24.287295 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:24.287262    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-77696685b-7mh27" event={"ID":"24555f40-5be1-4a70-9e33-ba3c95c1e02f","Type":"ContainerStarted","Data":"b116fffedf327affe833a9275c49ac098cd9e67e409d40ff588aa1dd3e981847"}
Aug 04 18:00:24.293769 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:24.293743    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-65bdbbff45-qjxkc" event={"ID":"1f245728-8f1a-43fe-9c0a-dde00647c750","Type":"ContainerStarted","Data":"3b14a9a4162ce9f6c114b5cc62e2e82546857a1032b14a31e2b04711a5fb3d68"}
Aug 04 18:00:24.345651 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:24.345614    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/6d05efcb-b6fb-485c-b472-bd406b8e729f-nginx-conf\") pod \"networking-console-plugin-f77f48c6d-lt6n7\" (UID: \"6d05efcb-b6fb-485c-b472-bd406b8e729f\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-lt6n7"
Aug 04 18:00:24.345802 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:24.345677    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6d05efcb-b6fb-485c-b472-bd406b8e729f-networking-console-plugin-cert\") pod \"networking-console-plugin-f77f48c6d-lt6n7\" (UID: \"6d05efcb-b6fb-485c-b472-bd406b8e729f\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-lt6n7"
Aug 04 18:00:24.345963 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:00:24.345937    2583 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Aug 04 18:00:24.346029 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:00:24.346006    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6d05efcb-b6fb-485c-b472-bd406b8e729f-networking-console-plugin-cert podName:6d05efcb-b6fb-485c-b472-bd406b8e729f nodeName:}" failed. No retries permitted until 2026-08-04 18:00:24.845987469 +0000 UTC m=+38.427206031 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/6d05efcb-b6fb-485c-b472-bd406b8e729f-networking-console-plugin-cert") pod "networking-console-plugin-f77f48c6d-lt6n7" (UID: "6d05efcb-b6fb-485c-b472-bd406b8e729f") : secret "networking-console-plugin-cert" not found
Aug 04 18:00:24.346912 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:24.346886    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/6d05efcb-b6fb-485c-b472-bd406b8e729f-nginx-conf\") pod \"networking-console-plugin-f77f48c6d-lt6n7\" (UID: \"6d05efcb-b6fb-485c-b472-bd406b8e729f\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-lt6n7"
Aug 04 18:00:24.849774 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:24.849734    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6d05efcb-b6fb-485c-b472-bd406b8e729f-networking-console-plugin-cert\") pod \"networking-console-plugin-f77f48c6d-lt6n7\" (UID: \"6d05efcb-b6fb-485c-b472-bd406b8e729f\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-lt6n7"
Aug 04 18:00:24.850245 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:00:24.849900    2583 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Aug 04 18:00:24.850245 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:00:24.849954    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6d05efcb-b6fb-485c-b472-bd406b8e729f-networking-console-plugin-cert podName:6d05efcb-b6fb-485c-b472-bd406b8e729f nodeName:}" failed. No retries permitted until 2026-08-04 18:00:25.849939516 +0000 UTC m=+39.431158072 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/6d05efcb-b6fb-485c-b472-bd406b8e729f-networking-console-plugin-cert") pod "networking-console-plugin-f77f48c6d-lt6n7" (UID: "6d05efcb-b6fb-485c-b472-bd406b8e729f") : secret "networking-console-plugin-cert" not found
Aug 04 18:00:25.300703 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:25.300407    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-v7q5c" event={"ID":"6b6fb1c8-d652-4046-8f8e-4f85375d5d08","Type":"ContainerStarted","Data":"9cbad5b8d21b767cf850cf3c6e23a932829dedf63833328a3e85e90060d74948"}
Aug 04 18:00:25.305628 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:25.305594    2583 generic.go:356] "Generic (PLEG): container finished" podID="555d14c0-e9e6-4119-8298-a88258e7c516" containerID="897de4b2793d104b146fcc8c67044bcf1cd865c8c26d5861fc8bfb2596fc7b06" exitCode=0
Aug 04 18:00:25.306304 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:25.306248    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-c6m8x" event={"ID":"555d14c0-e9e6-4119-8298-a88258e7c516","Type":"ContainerDied","Data":"897de4b2793d104b146fcc8c67044bcf1cd865c8c26d5861fc8bfb2596fc7b06"}
Aug 04 18:00:25.344876 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:25.344404    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-v7q5c" podStartSLOduration=13.090904347 podStartE2EDuration="14.344382192s" totalImagesPullingTime="1.253477845s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:00:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:00:25.319216269 +0000 UTC m=+38.900434847" watchObservedRunningTime="2026-08-04 18:00:25.344382192 +0000 UTC m=+38.925600774"
Aug 04 18:00:25.559869 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:25.559789    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/88e72b20-e499-46f0-9374-258bf49f39c2-service-ca-bundle\") pod \"router-default-85c4d8f7fd-bm9rl\" (UID: \"88e72b20-e499-46f0-9374-258bf49f39c2\") " pod="openshift-ingress/router-default-85c4d8f7fd-bm9rl"
Aug 04 18:00:25.560010 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:25.559873    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/88e72b20-e499-46f0-9374-258bf49f39c2-metrics-certs\") pod \"router-default-85c4d8f7fd-bm9rl\" (UID: \"88e72b20-e499-46f0-9374-258bf49f39c2\") " pod="openshift-ingress/router-default-85c4d8f7fd-bm9rl"
Aug 04 18:00:25.560010 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:25.559901    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4704d0e3-b459-422d-92d9-0e9ec9bbe24f-registry-tls\") pod \"image-registry-74cb995cd-pjjzg\" (UID: \"4704d0e3-b459-422d-92d9-0e9ec9bbe24f\") " pod="openshift-image-registry/image-registry-74cb995cd-pjjzg"
Aug 04 18:00:25.560010 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:25.559963    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3fb60b38-9cb9-4a07-9334-1b914ddd4a62-metrics-tls\") pod \"dns-default-7wnlf\" (UID: \"3fb60b38-9cb9-4a07-9334-1b914ddd4a62\") " pod="openshift-dns/dns-default-7wnlf"
Aug 04 18:00:25.560010 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:00:25.559983    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/88e72b20-e499-46f0-9374-258bf49f39c2-service-ca-bundle podName:88e72b20-e499-46f0-9374-258bf49f39c2 nodeName:}" failed. No retries permitted until 2026-08-04 18:00:29.559960746 +0000 UTC m=+43.141179316 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/88e72b20-e499-46f0-9374-258bf49f39c2-service-ca-bundle") pod "router-default-85c4d8f7fd-bm9rl" (UID: "88e72b20-e499-46f0-9374-258bf49f39c2") : configmap references non-existent config key: service-ca.crt
Aug 04 18:00:25.560245 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:25.560030    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7ef28a34-7ec8-49f8-a58c-920661139d9c-samples-operator-tls\") pod \"cluster-samples-operator-559cbf5745-qqg4c\" (UID: \"7ef28a34-7ec8-49f8-a58c-920661139d9c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-qqg4c"
Aug 04 18:00:25.560245 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:00:25.560058    2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Aug 04 18:00:25.560245 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:25.560074    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e079c95a-25bd-44e2-8f30-44d6cba4add7-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6f55c4f58d-fzk78\" (UID: \"e079c95a-25bd-44e2-8f30-44d6cba4add7\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-fzk78"
Aug 04 18:00:25.560245 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:00:25.560102    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3fb60b38-9cb9-4a07-9334-1b914ddd4a62-metrics-tls podName:3fb60b38-9cb9-4a07-9334-1b914ddd4a62 nodeName:}" failed. No retries permitted until 2026-08-04 18:00:29.560087447 +0000 UTC m=+43.141306010 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3fb60b38-9cb9-4a07-9334-1b914ddd4a62-metrics-tls") pod "dns-default-7wnlf" (UID: "3fb60b38-9cb9-4a07-9334-1b914ddd4a62") : secret "dns-default-metrics-tls" not found
Aug 04 18:00:25.560245 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:00:25.560152    2583 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Aug 04 18:00:25.560245 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:00:25.560187    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e079c95a-25bd-44e2-8f30-44d6cba4add7-cluster-monitoring-operator-tls podName:e079c95a-25bd-44e2-8f30-44d6cba4add7 nodeName:}" failed. No retries permitted until 2026-08-04 18:00:29.560176395 +0000 UTC m=+43.141394950 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/e079c95a-25bd-44e2-8f30-44d6cba4add7-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6f55c4f58d-fzk78" (UID: "e079c95a-25bd-44e2-8f30-44d6cba4add7") : secret "cluster-monitoring-operator-tls" not found
Aug 04 18:00:25.560245 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:00:25.560200    2583 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Aug 04 18:00:25.560245 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:00:25.560234    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/88e72b20-e499-46f0-9374-258bf49f39c2-metrics-certs podName:88e72b20-e499-46f0-9374-258bf49f39c2 nodeName:}" failed. No retries permitted until 2026-08-04 18:00:29.560222261 +0000 UTC m=+43.141440817 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/88e72b20-e499-46f0-9374-258bf49f39c2-metrics-certs") pod "router-default-85c4d8f7fd-bm9rl" (UID: "88e72b20-e499-46f0-9374-258bf49f39c2") : secret "router-metrics-certs-default" not found
Aug 04 18:00:25.560667 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:00:25.560249    2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Aug 04 18:00:25.560667 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:00:25.560264    2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-74cb995cd-pjjzg: secret "image-registry-tls" not found
Aug 04 18:00:25.560667 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:00:25.560288    2583 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Aug 04 18:00:25.560667 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:00:25.560301    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4704d0e3-b459-422d-92d9-0e9ec9bbe24f-registry-tls podName:4704d0e3-b459-422d-92d9-0e9ec9bbe24f nodeName:}" failed. No retries permitted until 2026-08-04 18:00:29.560288402 +0000 UTC m=+43.141506979 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4704d0e3-b459-422d-92d9-0e9ec9bbe24f-registry-tls") pod "image-registry-74cb995cd-pjjzg" (UID: "4704d0e3-b459-422d-92d9-0e9ec9bbe24f") : secret "image-registry-tls" not found
Aug 04 18:00:25.560667 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:00:25.560319    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ef28a34-7ec8-49f8-a58c-920661139d9c-samples-operator-tls podName:7ef28a34-7ec8-49f8-a58c-920661139d9c nodeName:}" failed. No retries permitted until 2026-08-04 18:00:29.560308712 +0000 UTC m=+43.141527271 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/7ef28a34-7ec8-49f8-a58c-920661139d9c-samples-operator-tls") pod "cluster-samples-operator-559cbf5745-qqg4c" (UID: "7ef28a34-7ec8-49f8-a58c-920661139d9c") : secret "samples-operator-tls" not found
Aug 04 18:00:25.863921 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:25.863228    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6d05efcb-b6fb-485c-b472-bd406b8e729f-networking-console-plugin-cert\") pod \"networking-console-plugin-f77f48c6d-lt6n7\" (UID: \"6d05efcb-b6fb-485c-b472-bd406b8e729f\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-lt6n7"
Aug 04 18:00:25.863921 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:00:25.863517    2583 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Aug 04 18:00:25.863921 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:00:25.863585    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6d05efcb-b6fb-485c-b472-bd406b8e729f-networking-console-plugin-cert podName:6d05efcb-b6fb-485c-b472-bd406b8e729f nodeName:}" failed. No retries permitted until 2026-08-04 18:00:27.863565779 +0000 UTC m=+41.444784337 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/6d05efcb-b6fb-485c-b472-bd406b8e729f-networking-console-plugin-cert") pod "networking-console-plugin-f77f48c6d-lt6n7" (UID: "6d05efcb-b6fb-485c-b472-bd406b8e729f") : secret "networking-console-plugin-cert" not found
Aug 04 18:00:26.315448 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:26.315410    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-c6m8x" event={"ID":"555d14c0-e9e6-4119-8298-a88258e7c516","Type":"ContainerStarted","Data":"df6dd25a86c8738983185dc11c3ca393ef1d6d26a84c252b49b1cfd9ae63ada5"}
Aug 04 18:00:26.344533 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:26.344441    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-c6m8x" podStartSLOduration=7.877433762 podStartE2EDuration="39.344419347s" totalImagesPullingTime="31.466985585s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 17:59:47 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:00:26.342040572 +0000 UTC m=+39.923259150" watchObservedRunningTime="2026-08-04 18:00:26.344419347 +0000 UTC m=+39.925637927"
Aug 04 18:00:27.883490 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:27.883441    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6d05efcb-b6fb-485c-b472-bd406b8e729f-networking-console-plugin-cert\") pod \"networking-console-plugin-f77f48c6d-lt6n7\" (UID: \"6d05efcb-b6fb-485c-b472-bd406b8e729f\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-lt6n7"
Aug 04 18:00:27.883935 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:00:27.883612    2583 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Aug 04 18:00:27.883935 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:00:27.883684    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6d05efcb-b6fb-485c-b472-bd406b8e729f-networking-console-plugin-cert podName:6d05efcb-b6fb-485c-b472-bd406b8e729f nodeName:}" failed. No retries permitted until 2026-08-04 18:00:31.883663035 +0000 UTC m=+45.464881591 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/6d05efcb-b6fb-485c-b472-bd406b8e729f-networking-console-plugin-cert") pod "networking-console-plugin-f77f48c6d-lt6n7" (UID: "6d05efcb-b6fb-485c-b472-bd406b8e729f") : secret "networking-console-plugin-cert" not found
Aug 04 18:00:29.597534 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:29.597494    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/88e72b20-e499-46f0-9374-258bf49f39c2-service-ca-bundle\") pod \"router-default-85c4d8f7fd-bm9rl\" (UID: \"88e72b20-e499-46f0-9374-258bf49f39c2\") " pod="openshift-ingress/router-default-85c4d8f7fd-bm9rl"
Aug 04 18:00:29.597974 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:29.597591    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/88e72b20-e499-46f0-9374-258bf49f39c2-metrics-certs\") pod \"router-default-85c4d8f7fd-bm9rl\" (UID: \"88e72b20-e499-46f0-9374-258bf49f39c2\") " pod="openshift-ingress/router-default-85c4d8f7fd-bm9rl"
Aug 04 18:00:29.597974 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:29.597618    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4704d0e3-b459-422d-92d9-0e9ec9bbe24f-registry-tls\") pod \"image-registry-74cb995cd-pjjzg\" (UID: \"4704d0e3-b459-422d-92d9-0e9ec9bbe24f\") " pod="openshift-image-registry/image-registry-74cb995cd-pjjzg"
Aug 04 18:00:29.597974 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:00:29.597670    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/88e72b20-e499-46f0-9374-258bf49f39c2-service-ca-bundle podName:88e72b20-e499-46f0-9374-258bf49f39c2 nodeName:}" failed. No retries permitted until 2026-08-04 18:00:37.59765063 +0000 UTC m=+51.178869185 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/88e72b20-e499-46f0-9374-258bf49f39c2-service-ca-bundle") pod "router-default-85c4d8f7fd-bm9rl" (UID: "88e72b20-e499-46f0-9374-258bf49f39c2") : configmap references non-existent config key: service-ca.crt
Aug 04 18:00:29.597974 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:29.597721    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3fb60b38-9cb9-4a07-9334-1b914ddd4a62-metrics-tls\") pod \"dns-default-7wnlf\" (UID: \"3fb60b38-9cb9-4a07-9334-1b914ddd4a62\") " pod="openshift-dns/dns-default-7wnlf"
Aug 04 18:00:29.597974 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:00:29.597743    2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Aug 04 18:00:29.597974 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:00:29.597758    2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-74cb995cd-pjjzg: secret "image-registry-tls" not found
Aug 04 18:00:29.597974 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:00:29.597792    2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Aug 04 18:00:29.597974 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:00:29.597808    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4704d0e3-b459-422d-92d9-0e9ec9bbe24f-registry-tls podName:4704d0e3-b459-422d-92d9-0e9ec9bbe24f nodeName:}" failed. No retries permitted until 2026-08-04 18:00:37.597792584 +0000 UTC m=+51.179011165 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4704d0e3-b459-422d-92d9-0e9ec9bbe24f-registry-tls") pod "image-registry-74cb995cd-pjjzg" (UID: "4704d0e3-b459-422d-92d9-0e9ec9bbe24f") : secret "image-registry-tls" not found
Aug 04 18:00:29.597974 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:29.597754    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7ef28a34-7ec8-49f8-a58c-920661139d9c-samples-operator-tls\") pod \"cluster-samples-operator-559cbf5745-qqg4c\" (UID: \"7ef28a34-7ec8-49f8-a58c-920661139d9c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-qqg4c"
Aug 04 18:00:29.597974 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:00:29.597822    2583 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Aug 04 18:00:29.597974 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:00:29.597741    2583 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Aug 04 18:00:29.597974 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:00:29.597844    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3fb60b38-9cb9-4a07-9334-1b914ddd4a62-metrics-tls podName:3fb60b38-9cb9-4a07-9334-1b914ddd4a62 nodeName:}" failed. No retries permitted until 2026-08-04 18:00:37.597829501 +0000 UTC m=+51.179048059 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3fb60b38-9cb9-4a07-9334-1b914ddd4a62-metrics-tls") pod "dns-default-7wnlf" (UID: "3fb60b38-9cb9-4a07-9334-1b914ddd4a62") : secret "dns-default-metrics-tls" not found
Aug 04 18:00:29.597974 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:00:29.597869    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ef28a34-7ec8-49f8-a58c-920661139d9c-samples-operator-tls podName:7ef28a34-7ec8-49f8-a58c-920661139d9c nodeName:}" failed. No retries permitted until 2026-08-04 18:00:37.597860273 +0000 UTC m=+51.179078834 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/7ef28a34-7ec8-49f8-a58c-920661139d9c-samples-operator-tls") pod "cluster-samples-operator-559cbf5745-qqg4c" (UID: "7ef28a34-7ec8-49f8-a58c-920661139d9c") : secret "samples-operator-tls" not found
Aug 04 18:00:29.597974 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:29.597905    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e079c95a-25bd-44e2-8f30-44d6cba4add7-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6f55c4f58d-fzk78\" (UID: \"e079c95a-25bd-44e2-8f30-44d6cba4add7\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-fzk78"
Aug 04 18:00:29.597974 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:00:29.597923    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/88e72b20-e499-46f0-9374-258bf49f39c2-metrics-certs podName:88e72b20-e499-46f0-9374-258bf49f39c2 nodeName:}" failed. No retries permitted until 2026-08-04 18:00:37.597900118 +0000 UTC m=+51.179118687 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/88e72b20-e499-46f0-9374-258bf49f39c2-metrics-certs") pod "router-default-85c4d8f7fd-bm9rl" (UID: "88e72b20-e499-46f0-9374-258bf49f39c2") : secret "router-metrics-certs-default" not found
Aug 04 18:00:29.597974 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:00:29.597970    2583 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Aug 04 18:00:29.598539 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:00:29.598013    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e079c95a-25bd-44e2-8f30-44d6cba4add7-cluster-monitoring-operator-tls podName:e079c95a-25bd-44e2-8f30-44d6cba4add7 nodeName:}" failed. No retries permitted until 2026-08-04 18:00:37.598001852 +0000 UTC m=+51.179220411 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/e079c95a-25bd-44e2-8f30-44d6cba4add7-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6f55c4f58d-fzk78" (UID: "e079c95a-25bd-44e2-8f30-44d6cba4add7") : secret "cluster-monitoring-operator-tls" not found
Aug 04 18:00:31.916617 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:31.916536    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6d05efcb-b6fb-485c-b472-bd406b8e729f-networking-console-plugin-cert\") pod \"networking-console-plugin-f77f48c6d-lt6n7\" (UID: \"6d05efcb-b6fb-485c-b472-bd406b8e729f\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-lt6n7"
Aug 04 18:00:31.917116 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:00:31.916700    2583 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Aug 04 18:00:31.917116 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:00:31.916774    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6d05efcb-b6fb-485c-b472-bd406b8e729f-networking-console-plugin-cert podName:6d05efcb-b6fb-485c-b472-bd406b8e729f nodeName:}" failed. No retries permitted until 2026-08-04 18:00:39.916753538 +0000 UTC m=+53.497972110 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/6d05efcb-b6fb-485c-b472-bd406b8e729f-networking-console-plugin-cert") pod "networking-console-plugin-f77f48c6d-lt6n7" (UID: "6d05efcb-b6fb-485c-b472-bd406b8e729f") : secret "networking-console-plugin-cert" not found
Aug 04 18:00:32.332953 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:32.332927    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-wskfb_6157c2d3-2fd0-4544-88ad-434ac75f8c04/console-operator/0.log"
Aug 04 18:00:32.333195 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:32.332965    2583 generic.go:356] "Generic (PLEG): container finished" podID="6157c2d3-2fd0-4544-88ad-434ac75f8c04" containerID="bd402c54e5be89c3b448bdd33225fccd1b902aa5fd38c63649cc70b00b7c71d2" exitCode=255
Aug 04 18:00:32.333195 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:32.333048    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-5ccd6b445f-wskfb" event={"ID":"6157c2d3-2fd0-4544-88ad-434ac75f8c04","Type":"ContainerDied","Data":"bd402c54e5be89c3b448bdd33225fccd1b902aa5fd38c63649cc70b00b7c71d2"}
Aug 04 18:00:32.333395 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:32.333370    2583 scope.go:117] "RemoveContainer" containerID="bd402c54e5be89c3b448bdd33225fccd1b902aa5fd38c63649cc70b00b7c71d2"
Aug 04 18:00:32.336796 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:32.336757    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-77696685b-7mh27" event={"ID":"24555f40-5be1-4a70-9e33-ba3c95c1e02f","Type":"ContainerStarted","Data":"735b60adf9b4da8976cbd6812dabcf77eaa1cdb45e4122579fe560862bad60b4"}
Aug 04 18:00:32.338483 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:32.338449    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-65bdbbff45-qjxkc" event={"ID":"1f245728-8f1a-43fe-9c0a-dde00647c750","Type":"ContainerStarted","Data":"1d66716fc92461de4a7d28c392729dd9d4b1a28074a86d7f7a724a036d6d718a"}
Aug 04 18:00:32.340113 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:32.340090    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-jwhg9" event={"ID":"32978e3c-1e6b-40dc-9dd9-f5bb8f9777ef","Type":"ContainerStarted","Data":"3a98dabd45cd39258c326861f458d29b4998cb96a0816d8c81a6c2c198c0f343"}
Aug 04 18:00:32.341697 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:32.341672    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-vlvdv" event={"ID":"f46d23da-d73f-4ada-8ac2-1a0e3313e664","Type":"ContainerStarted","Data":"d99eda6ca4f5cac6bffca2e0cc67a5fd3904d5d1673576497ad87f58447c3142"}
Aug 04 18:00:32.343102 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:32.343082    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-6m4nc" event={"ID":"582f0d25-f53a-491b-ae73-d09ed1ab0501","Type":"ContainerStarted","Data":"a3b840c76b51b2faac32c58329c4188e07b82a92cccdec786b08d19af4e0444e"}
Aug 04 18:00:32.373174 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:32.373128    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-6m4nc" podStartSLOduration=13.718570059 podStartE2EDuration="21.373111931s" totalImagesPullingTime="7.654541872s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:00:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:00:32.371298522 +0000 UTC m=+45.952517114" watchObservedRunningTime="2026-08-04 18:00:32.373111931 +0000 UTC m=+45.954330509"
Aug 04 18:00:32.393049 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:32.391557    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-jwhg9" podStartSLOduration=13.732934878 podStartE2EDuration="21.391542987s" totalImagesPullingTime="7.658608109s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:00:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:00:32.390008091 +0000 UTC m=+45.971226670" watchObservedRunningTime="2026-08-04 18:00:32.391542987 +0000 UTC m=+45.972761568"
Aug 04 18:00:32.410677 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:32.410622    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-65bdbbff45-qjxkc" podStartSLOduration=13.75573249 podStartE2EDuration="21.410606076s" totalImagesPullingTime="7.654873586s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:00:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:00:32.409371375 +0000 UTC m=+45.990589954" watchObservedRunningTime="2026-08-04 18:00:32.410606076 +0000 UTC m=+45.991824638"
Aug 04 18:00:32.435326 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:32.430533    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-vlvdv" podStartSLOduration=3.898691607 podStartE2EDuration="11.430515529s" totalImagesPullingTime="7.531823922s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:00:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:00:32.427195225 +0000 UTC m=+46.008413796" watchObservedRunningTime="2026-08-04 18:00:32.430515529 +0000 UTC m=+46.011734108"
Aug 04 18:00:33.056036 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:33.055377    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-77696685b-7mh27" podStartSLOduration=14.395255204 podStartE2EDuration="22.055351396s" totalImagesPullingTime="7.660096192s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:00:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:00:32.447209458 +0000 UTC m=+46.028428036" watchObservedRunningTime="2026-08-04 18:00:33.055351396 +0000 UTC m=+46.636569977"
Aug 04 18:00:33.056863 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:33.056836    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-xzw5k"]
Aug 04 18:00:33.074723 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:33.074705    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-xzw5k"
Aug 04 18:00:33.077070 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:33.077011    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-xzw5k"]
Aug 04 18:00:33.077879 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:33.077697    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-sm4mb\""
Aug 04 18:00:33.078559 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:33.078055    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Aug 04 18:00:33.078559 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:33.078318    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Aug 04 18:00:33.128231 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:33.128195    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/31401da2-69b4-4679-8683-63aa4223f38b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xzw5k\" (UID: \"31401da2-69b4-4679-8683-63aa4223f38b\") " pod="openshift-insights/insights-runtime-extractor-xzw5k"
Aug 04 18:00:33.128231 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:33.128237    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/31401da2-69b4-4679-8683-63aa4223f38b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xzw5k\" (UID: \"31401da2-69b4-4679-8683-63aa4223f38b\") " pod="openshift-insights/insights-runtime-extractor-xzw5k"
Aug 04 18:00:33.128421 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:33.128401    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/31401da2-69b4-4679-8683-63aa4223f38b-data-volume\") pod \"insights-runtime-extractor-xzw5k\" (UID: \"31401da2-69b4-4679-8683-63aa4223f38b\") " pod="openshift-insights/insights-runtime-extractor-xzw5k"
Aug 04 18:00:33.128491 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:33.128440    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czf28\" (UniqueName: \"kubernetes.io/projected/31401da2-69b4-4679-8683-63aa4223f38b-kube-api-access-czf28\") pod \"insights-runtime-extractor-xzw5k\" (UID: \"31401da2-69b4-4679-8683-63aa4223f38b\") " pod="openshift-insights/insights-runtime-extractor-xzw5k"
Aug 04 18:00:33.128572 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:33.128558    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/31401da2-69b4-4679-8683-63aa4223f38b-crio-socket\") pod \"insights-runtime-extractor-xzw5k\" (UID: \"31401da2-69b4-4679-8683-63aa4223f38b\") " pod="openshift-insights/insights-runtime-extractor-xzw5k"
Aug 04 18:00:33.229460 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:33.229432    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/31401da2-69b4-4679-8683-63aa4223f38b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xzw5k\" (UID: \"31401da2-69b4-4679-8683-63aa4223f38b\") " pod="openshift-insights/insights-runtime-extractor-xzw5k"
Aug 04 18:00:33.229626 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:33.229463    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/31401da2-69b4-4679-8683-63aa4223f38b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xzw5k\" (UID: \"31401da2-69b4-4679-8683-63aa4223f38b\") " pod="openshift-insights/insights-runtime-extractor-xzw5k"
Aug 04 18:00:33.229626 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:00:33.229549    2583 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Aug 04 18:00:33.229626 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:33.229579    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/31401da2-69b4-4679-8683-63aa4223f38b-data-volume\") pod \"insights-runtime-extractor-xzw5k\" (UID: \"31401da2-69b4-4679-8683-63aa4223f38b\") " pod="openshift-insights/insights-runtime-extractor-xzw5k"
Aug 04 18:00:33.229626 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:00:33.229606    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/31401da2-69b4-4679-8683-63aa4223f38b-kube-rbac-proxy-cm podName:31401da2-69b4-4679-8683-63aa4223f38b nodeName:}" failed. No retries permitted until 2026-08-04 18:00:33.729591039 +0000 UTC m=+47.310809598 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/31401da2-69b4-4679-8683-63aa4223f38b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-xzw5k" (UID: "31401da2-69b4-4679-8683-63aa4223f38b") : configmap "kube-rbac-proxy" not found
Aug 04 18:00:33.229626 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:33.229624    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czf28\" (UniqueName: \"kubernetes.io/projected/31401da2-69b4-4679-8683-63aa4223f38b-kube-api-access-czf28\") pod \"insights-runtime-extractor-xzw5k\" (UID: \"31401da2-69b4-4679-8683-63aa4223f38b\") " pod="openshift-insights/insights-runtime-extractor-xzw5k"
Aug 04 18:00:33.229881 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:33.229712    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/31401da2-69b4-4679-8683-63aa4223f38b-crio-socket\") pod \"insights-runtime-extractor-xzw5k\" (UID: \"31401da2-69b4-4679-8683-63aa4223f38b\") " pod="openshift-insights/insights-runtime-extractor-xzw5k"
Aug 04 18:00:33.229881 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:00:33.229720    2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Aug 04 18:00:33.229881 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:00:33.229778    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/31401da2-69b4-4679-8683-63aa4223f38b-insights-runtime-extractor-tls podName:31401da2-69b4-4679-8683-63aa4223f38b nodeName:}" failed. No retries permitted until 2026-08-04 18:00:33.729761867 +0000 UTC m=+47.310980427 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/31401da2-69b4-4679-8683-63aa4223f38b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xzw5k" (UID: "31401da2-69b4-4679-8683-63aa4223f38b") : secret "insights-runtime-extractor-tls" not found
Aug 04 18:00:33.230036 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:33.229902    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/31401da2-69b4-4679-8683-63aa4223f38b-data-volume\") pod \"insights-runtime-extractor-xzw5k\" (UID: \"31401da2-69b4-4679-8683-63aa4223f38b\") " pod="openshift-insights/insights-runtime-extractor-xzw5k"
Aug 04 18:00:33.230036 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:33.229924    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/31401da2-69b4-4679-8683-63aa4223f38b-crio-socket\") pod \"insights-runtime-extractor-xzw5k\" (UID: \"31401da2-69b4-4679-8683-63aa4223f38b\") " pod="openshift-insights/insights-runtime-extractor-xzw5k"
Aug 04 18:00:33.245445 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:33.245417    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-czf28\" (UniqueName: \"kubernetes.io/projected/31401da2-69b4-4679-8683-63aa4223f38b-kube-api-access-czf28\") pod \"insights-runtime-extractor-xzw5k\" (UID: \"31401da2-69b4-4679-8683-63aa4223f38b\") " pod="openshift-insights/insights-runtime-extractor-xzw5k"
Aug 04 18:00:33.348391 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:33.348328    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-wskfb_6157c2d3-2fd0-4544-88ad-434ac75f8c04/console-operator/1.log"
Aug 04 18:00:33.348812 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:33.348796    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-wskfb_6157c2d3-2fd0-4544-88ad-434ac75f8c04/console-operator/0.log"
Aug 04 18:00:33.348865 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:33.348830    2583 generic.go:356] "Generic (PLEG): container finished" podID="6157c2d3-2fd0-4544-88ad-434ac75f8c04" containerID="1d5a2cb734fa3629814b2e3d61648e52018126be65deed695e63a8b5505d2f25" exitCode=255
Aug 04 18:00:33.348924 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:33.348910    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-5ccd6b445f-wskfb" event={"ID":"6157c2d3-2fd0-4544-88ad-434ac75f8c04","Type":"ContainerDied","Data":"1d5a2cb734fa3629814b2e3d61648e52018126be65deed695e63a8b5505d2f25"}
Aug 04 18:00:33.348977 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:33.348953    2583 scope.go:117] "RemoveContainer" containerID="bd402c54e5be89c3b448bdd33225fccd1b902aa5fd38c63649cc70b00b7c71d2"
Aug 04 18:00:33.349186 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:33.349164    2583 scope.go:117] "RemoveContainer" containerID="1d5a2cb734fa3629814b2e3d61648e52018126be65deed695e63a8b5505d2f25"
Aug 04 18:00:33.349375 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:00:33.349354    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-5ccd6b445f-wskfb_openshift-console-operator(6157c2d3-2fd0-4544-88ad-434ac75f8c04)\"" pod="openshift-console-operator/console-operator-5ccd6b445f-wskfb" podUID="6157c2d3-2fd0-4544-88ad-434ac75f8c04"
Aug 04 18:00:33.735017 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:33.734989    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/31401da2-69b4-4679-8683-63aa4223f38b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xzw5k\" (UID: \"31401da2-69b4-4679-8683-63aa4223f38b\") " pod="openshift-insights/insights-runtime-extractor-xzw5k"
Aug 04 18:00:33.735017 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:33.735018    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/31401da2-69b4-4679-8683-63aa4223f38b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xzw5k\" (UID: \"31401da2-69b4-4679-8683-63aa4223f38b\") " pod="openshift-insights/insights-runtime-extractor-xzw5k"
Aug 04 18:00:33.735213 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:00:33.735107    2583 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Aug 04 18:00:33.735213 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:00:33.735176    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/31401da2-69b4-4679-8683-63aa4223f38b-kube-rbac-proxy-cm podName:31401da2-69b4-4679-8683-63aa4223f38b nodeName:}" failed. No retries permitted until 2026-08-04 18:00:34.735163435 +0000 UTC m=+48.316381991 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/31401da2-69b4-4679-8683-63aa4223f38b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-xzw5k" (UID: "31401da2-69b4-4679-8683-63aa4223f38b") : configmap "kube-rbac-proxy" not found
Aug 04 18:00:33.735213 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:00:33.735203    2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Aug 04 18:00:33.735342 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:00:33.735270    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/31401da2-69b4-4679-8683-63aa4223f38b-insights-runtime-extractor-tls podName:31401da2-69b4-4679-8683-63aa4223f38b nodeName:}" failed. No retries permitted until 2026-08-04 18:00:34.735257242 +0000 UTC m=+48.316475798 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/31401da2-69b4-4679-8683-63aa4223f38b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xzw5k" (UID: "31401da2-69b4-4679-8683-63aa4223f38b") : secret "insights-runtime-extractor-tls" not found
Aug 04 18:00:34.353510 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:34.353462    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-wskfb_6157c2d3-2fd0-4544-88ad-434ac75f8c04/console-operator/1.log"
Aug 04 18:00:34.353885 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:34.353791    2583 scope.go:117] "RemoveContainer" containerID="1d5a2cb734fa3629814b2e3d61648e52018126be65deed695e63a8b5505d2f25"
Aug 04 18:00:34.353946 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:00:34.353927    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-5ccd6b445f-wskfb_openshift-console-operator(6157c2d3-2fd0-4544-88ad-434ac75f8c04)\"" pod="openshift-console-operator/console-operator-5ccd6b445f-wskfb" podUID="6157c2d3-2fd0-4544-88ad-434ac75f8c04"
Aug 04 18:00:34.402699 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:34.402676    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-ggrqd_9fffa81f-5a52-4425-b941-1693ba961fba/dns-node-resolver/0.log"
Aug 04 18:00:34.742704 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:34.742673    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/31401da2-69b4-4679-8683-63aa4223f38b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xzw5k\" (UID: \"31401da2-69b4-4679-8683-63aa4223f38b\") " pod="openshift-insights/insights-runtime-extractor-xzw5k"
Aug 04 18:00:34.742704 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:34.742704    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/31401da2-69b4-4679-8683-63aa4223f38b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xzw5k\" (UID: \"31401da2-69b4-4679-8683-63aa4223f38b\") " pod="openshift-insights/insights-runtime-extractor-xzw5k"
Aug 04 18:00:34.742893 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:00:34.742794    2583 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Aug 04 18:00:34.742893 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:00:34.742862    2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Aug 04 18:00:34.742893 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:00:34.742865    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/31401da2-69b4-4679-8683-63aa4223f38b-kube-rbac-proxy-cm podName:31401da2-69b4-4679-8683-63aa4223f38b nodeName:}" failed. No retries permitted until 2026-08-04 18:00:36.742845804 +0000 UTC m=+50.324064379 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/31401da2-69b4-4679-8683-63aa4223f38b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-xzw5k" (UID: "31401da2-69b4-4679-8683-63aa4223f38b") : configmap "kube-rbac-proxy" not found
Aug 04 18:00:34.742993 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:00:34.742907    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/31401da2-69b4-4679-8683-63aa4223f38b-insights-runtime-extractor-tls podName:31401da2-69b4-4679-8683-63aa4223f38b nodeName:}" failed. No retries permitted until 2026-08-04 18:00:36.742895678 +0000 UTC m=+50.324114234 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/31401da2-69b4-4679-8683-63aa4223f38b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xzw5k" (UID: "31401da2-69b4-4679-8683-63aa4223f38b") : secret "insights-runtime-extractor-tls" not found
Aug 04 18:00:35.602955 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:35.602925    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-nz5rh_00bf4681-2beb-4489-aba4-7e85f7bf2aaa/node-ca/0.log"
Aug 04 18:00:36.406197 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:36.406171    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-vlvdv_f46d23da-d73f-4ada-8ac2-1a0e3313e664/serve-healthcheck-canary/0.log"
Aug 04 18:00:36.760553 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:36.760519    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/31401da2-69b4-4679-8683-63aa4223f38b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xzw5k\" (UID: \"31401da2-69b4-4679-8683-63aa4223f38b\") " pod="openshift-insights/insights-runtime-extractor-xzw5k"
Aug 04 18:00:36.760553 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:36.760555    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/31401da2-69b4-4679-8683-63aa4223f38b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xzw5k\" (UID: \"31401da2-69b4-4679-8683-63aa4223f38b\") " pod="openshift-insights/insights-runtime-extractor-xzw5k"
Aug 04 18:00:36.760980 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:00:36.760637    2583 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Aug 04 18:00:36.760980 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:00:36.760694    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/31401da2-69b4-4679-8683-63aa4223f38b-kube-rbac-proxy-cm podName:31401da2-69b4-4679-8683-63aa4223f38b nodeName:}" failed. No retries permitted until 2026-08-04 18:00:40.7606798 +0000 UTC m=+54.341898370 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/31401da2-69b4-4679-8683-63aa4223f38b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-xzw5k" (UID: "31401da2-69b4-4679-8683-63aa4223f38b") : configmap "kube-rbac-proxy" not found
Aug 04 18:00:36.760980 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:00:36.760730    2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Aug 04 18:00:36.760980 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:00:36.760779    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/31401da2-69b4-4679-8683-63aa4223f38b-insights-runtime-extractor-tls podName:31401da2-69b4-4679-8683-63aa4223f38b nodeName:}" failed. No retries permitted until 2026-08-04 18:00:40.760766884 +0000 UTC m=+54.341985439 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/31401da2-69b4-4679-8683-63aa4223f38b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xzw5k" (UID: "31401da2-69b4-4679-8683-63aa4223f38b") : secret "insights-runtime-extractor-tls" not found
Aug 04 18:00:37.667693 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:37.667661    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/88e72b20-e499-46f0-9374-258bf49f39c2-metrics-certs\") pod \"router-default-85c4d8f7fd-bm9rl\" (UID: \"88e72b20-e499-46f0-9374-258bf49f39c2\") " pod="openshift-ingress/router-default-85c4d8f7fd-bm9rl"
Aug 04 18:00:37.667884 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:37.667700    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4704d0e3-b459-422d-92d9-0e9ec9bbe24f-registry-tls\") pod \"image-registry-74cb995cd-pjjzg\" (UID: \"4704d0e3-b459-422d-92d9-0e9ec9bbe24f\") " pod="openshift-image-registry/image-registry-74cb995cd-pjjzg"
Aug 04 18:00:37.667884 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:37.667738    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3fb60b38-9cb9-4a07-9334-1b914ddd4a62-metrics-tls\") pod \"dns-default-7wnlf\" (UID: \"3fb60b38-9cb9-4a07-9334-1b914ddd4a62\") " pod="openshift-dns/dns-default-7wnlf"
Aug 04 18:00:37.667884 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:37.667763    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7ef28a34-7ec8-49f8-a58c-920661139d9c-samples-operator-tls\") pod \"cluster-samples-operator-559cbf5745-qqg4c\" (UID: \"7ef28a34-7ec8-49f8-a58c-920661139d9c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-qqg4c"
Aug 04 18:00:37.667884 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:37.667799    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e079c95a-25bd-44e2-8f30-44d6cba4add7-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6f55c4f58d-fzk78\" (UID: \"e079c95a-25bd-44e2-8f30-44d6cba4add7\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-fzk78"
Aug 04 18:00:37.667884 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:00:37.667804    2583 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Aug 04 18:00:37.667884 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:00:37.667853    2583 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Aug 04 18:00:37.667884 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:00:37.667872    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/88e72b20-e499-46f0-9374-258bf49f39c2-metrics-certs podName:88e72b20-e499-46f0-9374-258bf49f39c2 nodeName:}" failed. No retries permitted until 2026-08-04 18:00:53.667853391 +0000 UTC m=+67.249071951 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/88e72b20-e499-46f0-9374-258bf49f39c2-metrics-certs") pod "router-default-85c4d8f7fd-bm9rl" (UID: "88e72b20-e499-46f0-9374-258bf49f39c2") : secret "router-metrics-certs-default" not found
Aug 04 18:00:37.667884 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:00:37.667877    2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Aug 04 18:00:37.668192 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:00:37.667899    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ef28a34-7ec8-49f8-a58c-920661139d9c-samples-operator-tls podName:7ef28a34-7ec8-49f8-a58c-920661139d9c nodeName:}" failed. No retries permitted until 2026-08-04 18:00:53.667887183 +0000 UTC m=+67.249105744 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/7ef28a34-7ec8-49f8-a58c-920661139d9c-samples-operator-tls") pod "cluster-samples-operator-559cbf5745-qqg4c" (UID: "7ef28a34-7ec8-49f8-a58c-920661139d9c") : secret "samples-operator-tls" not found
Aug 04 18:00:37.668192 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:00:37.667810    2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Aug 04 18:00:37.668192 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:00:37.667917    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3fb60b38-9cb9-4a07-9334-1b914ddd4a62-metrics-tls podName:3fb60b38-9cb9-4a07-9334-1b914ddd4a62 nodeName:}" failed. No retries permitted until 2026-08-04 18:00:53.66790661 +0000 UTC m=+67.249125166 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3fb60b38-9cb9-4a07-9334-1b914ddd4a62-metrics-tls") pod "dns-default-7wnlf" (UID: "3fb60b38-9cb9-4a07-9334-1b914ddd4a62") : secret "dns-default-metrics-tls" not found
Aug 04 18:00:37.668192 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:00:37.667925    2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-74cb995cd-pjjzg: secret "image-registry-tls" not found
Aug 04 18:00:37.668192 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:37.667937    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/88e72b20-e499-46f0-9374-258bf49f39c2-service-ca-bundle\") pod \"router-default-85c4d8f7fd-bm9rl\" (UID: \"88e72b20-e499-46f0-9374-258bf49f39c2\") " pod="openshift-ingress/router-default-85c4d8f7fd-bm9rl"
Aug 04 18:00:37.668192 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:00:37.667954    2583 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Aug 04 18:00:37.668192 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:00:37.667974    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4704d0e3-b459-422d-92d9-0e9ec9bbe24f-registry-tls podName:4704d0e3-b459-422d-92d9-0e9ec9bbe24f nodeName:}" failed. No retries permitted until 2026-08-04 18:00:53.667958225 +0000 UTC m=+67.249176799 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4704d0e3-b459-422d-92d9-0e9ec9bbe24f-registry-tls") pod "image-registry-74cb995cd-pjjzg" (UID: "4704d0e3-b459-422d-92d9-0e9ec9bbe24f") : secret "image-registry-tls" not found
Aug 04 18:00:37.668192 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:00:37.668000    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e079c95a-25bd-44e2-8f30-44d6cba4add7-cluster-monitoring-operator-tls podName:e079c95a-25bd-44e2-8f30-44d6cba4add7 nodeName:}" failed. No retries permitted until 2026-08-04 18:00:53.667988164 +0000 UTC m=+67.249206725 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/e079c95a-25bd-44e2-8f30-44d6cba4add7-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6f55c4f58d-fzk78" (UID: "e079c95a-25bd-44e2-8f30-44d6cba4add7") : secret "cluster-monitoring-operator-tls" not found
Aug 04 18:00:37.668192 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:00:37.668017    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/88e72b20-e499-46f0-9374-258bf49f39c2-service-ca-bundle podName:88e72b20-e499-46f0-9374-258bf49f39c2 nodeName:}" failed. No retries permitted until 2026-08-04 18:00:53.668007588 +0000 UTC m=+67.249226151 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/88e72b20-e499-46f0-9374-258bf49f39c2-service-ca-bundle") pod "router-default-85c4d8f7fd-bm9rl" (UID: "88e72b20-e499-46f0-9374-258bf49f39c2") : configmap references non-existent config key: service-ca.crt
Aug 04 18:00:39.986806 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:39.986768    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6d05efcb-b6fb-485c-b472-bd406b8e729f-networking-console-plugin-cert\") pod \"networking-console-plugin-f77f48c6d-lt6n7\" (UID: \"6d05efcb-b6fb-485c-b472-bd406b8e729f\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-lt6n7"
Aug 04 18:00:39.987224 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:00:39.986914    2583 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Aug 04 18:00:39.987224 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:00:39.986980    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6d05efcb-b6fb-485c-b472-bd406b8e729f-networking-console-plugin-cert podName:6d05efcb-b6fb-485c-b472-bd406b8e729f nodeName:}" failed. No retries permitted until 2026-08-04 18:00:55.986965404 +0000 UTC m=+69.568183979 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/6d05efcb-b6fb-485c-b472-bd406b8e729f-networking-console-plugin-cert") pod "networking-console-plugin-f77f48c6d-lt6n7" (UID: "6d05efcb-b6fb-485c-b472-bd406b8e729f") : secret "networking-console-plugin-cert" not found
Aug 04 18:00:40.793898 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:40.793860    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/31401da2-69b4-4679-8683-63aa4223f38b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xzw5k\" (UID: \"31401da2-69b4-4679-8683-63aa4223f38b\") " pod="openshift-insights/insights-runtime-extractor-xzw5k"
Aug 04 18:00:40.794179 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:40.793903    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/31401da2-69b4-4679-8683-63aa4223f38b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xzw5k\" (UID: \"31401da2-69b4-4679-8683-63aa4223f38b\") " pod="openshift-insights/insights-runtime-extractor-xzw5k"
Aug 04 18:00:40.794179 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:00:40.794069    2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Aug 04 18:00:40.794179 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:00:40.794134    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/31401da2-69b4-4679-8683-63aa4223f38b-insights-runtime-extractor-tls podName:31401da2-69b4-4679-8683-63aa4223f38b nodeName:}" failed. No retries permitted until 2026-08-04 18:00:48.794117749 +0000 UTC m=+62.375336305 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/31401da2-69b4-4679-8683-63aa4223f38b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xzw5k" (UID: "31401da2-69b4-4679-8683-63aa4223f38b") : secret "insights-runtime-extractor-tls" not found
Aug 04 18:00:40.794365 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:40.794343    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/31401da2-69b4-4679-8683-63aa4223f38b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xzw5k\" (UID: \"31401da2-69b4-4679-8683-63aa4223f38b\") " pod="openshift-insights/insights-runtime-extractor-xzw5k"
Aug 04 18:00:42.132614 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:42.132583    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-5ccd6b445f-wskfb"
Aug 04 18:00:42.132969 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:42.132635    2583 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-5ccd6b445f-wskfb"
Aug 04 18:00:42.132969 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:42.132931    2583 scope.go:117] "RemoveContainer" containerID="1d5a2cb734fa3629814b2e3d61648e52018126be65deed695e63a8b5505d2f25"
Aug 04 18:00:42.133075 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:00:42.133056    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-5ccd6b445f-wskfb_openshift-console-operator(6157c2d3-2fd0-4544-88ad-434ac75f8c04)\"" pod="openshift-console-operator/console-operator-5ccd6b445f-wskfb" podUID="6157c2d3-2fd0-4544-88ad-434ac75f8c04"
Aug 04 18:00:47.261491 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:47.261448    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mzl4m"
Aug 04 18:00:48.858242 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:48.858207    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/31401da2-69b4-4679-8683-63aa4223f38b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xzw5k\" (UID: \"31401da2-69b4-4679-8683-63aa4223f38b\") " pod="openshift-insights/insights-runtime-extractor-xzw5k"
Aug 04 18:00:48.860613 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:48.860577    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/31401da2-69b4-4679-8683-63aa4223f38b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xzw5k\" (UID: \"31401da2-69b4-4679-8683-63aa4223f38b\") " pod="openshift-insights/insights-runtime-extractor-xzw5k"
Aug 04 18:00:48.989132 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:48.989107    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-sm4mb\""
Aug 04 18:00:48.996592 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:48.996571    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-xzw5k"
Aug 04 18:00:49.118037 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:49.118003    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-xzw5k"]
Aug 04 18:00:49.120381 ip-10-0-135-124 kubenswrapper[2583]: W0804 18:00:49.120360    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31401da2_69b4_4679_8683_63aa4223f38b.slice/crio-4390bcb7ad8d013cfe1e4ea093a8caa0bf077036059eba26b7104b13194ccec3 WatchSource:0}: Error finding container 4390bcb7ad8d013cfe1e4ea093a8caa0bf077036059eba26b7104b13194ccec3: Status 404 returned error can't find the container with id 4390bcb7ad8d013cfe1e4ea093a8caa0bf077036059eba26b7104b13194ccec3
Aug 04 18:00:49.398492 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:49.398401    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xzw5k" event={"ID":"31401da2-69b4-4679-8683-63aa4223f38b","Type":"ContainerStarted","Data":"42eac23e94cd33702c31dd707d52a217b1eeb59a3248ce1bd3f786d75d59eea7"}
Aug 04 18:00:49.398492 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:49.398456    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xzw5k" event={"ID":"31401da2-69b4-4679-8683-63aa4223f38b","Type":"ContainerStarted","Data":"4390bcb7ad8d013cfe1e4ea093a8caa0bf077036059eba26b7104b13194ccec3"}
Aug 04 18:00:50.407103 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:50.407027    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xzw5k" event={"ID":"31401da2-69b4-4679-8683-63aa4223f38b","Type":"ContainerStarted","Data":"6f055ce23800ca05bf799f5b2f452168b2069fb4b612405adc14a7d7f6f2cc4b"}
Aug 04 18:00:52.415120 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:52.415082    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xzw5k" event={"ID":"31401da2-69b4-4679-8683-63aa4223f38b","Type":"ContainerStarted","Data":"a5a9fe7011bccc3e7c639675cf6609d14b1116336a1c7fc4eb797819d7e81f06"}
Aug 04 18:00:52.439207 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:52.439157    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-xzw5k" podStartSLOduration=17.143560513 podStartE2EDuration="19.439143378s" totalImagesPullingTime="2.295582865s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 18:00:33 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:00:52.438781859 +0000 UTC m=+66.020000436" watchObservedRunningTime="2026-08-04 18:00:52.439143378 +0000 UTC m=+66.020361955"
Aug 04 18:00:52.690350 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:52.690322    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87rpt\" (UniqueName: \"kubernetes.io/projected/43697de6-c2bd-48c6-81eb-1f37fa5d3174-kube-api-access-87rpt\") pod \"network-check-target-jxnpj\" (UID: \"43697de6-c2bd-48c6-81eb-1f37fa5d3174\") " pod="openshift-network-diagnostics/network-check-target-jxnpj"
Aug 04 18:00:52.692795 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:52.692770    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-87rpt\" (UniqueName: \"kubernetes.io/projected/43697de6-c2bd-48c6-81eb-1f37fa5d3174-kube-api-access-87rpt\") pod \"network-check-target-jxnpj\" (UID: \"43697de6-c2bd-48c6-81eb-1f37fa5d3174\") " pod="openshift-network-diagnostics/network-check-target-jxnpj"
Aug 04 18:00:52.791389 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:52.791354    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/31fd88d9-8ce3-4056-a185-4427ec8939c1-original-pull-secret\") pod \"global-pull-secret-syncer-lqxxl\" (UID: \"31fd88d9-8ce3-4056-a185-4427ec8939c1\") " pod="kube-system/global-pull-secret-syncer-lqxxl"
Aug 04 18:00:52.791543 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:52.791401    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0480eb75-3ca1-46ba-9606-ec9e98dc9c4c-metrics-certs\") pod \"network-metrics-daemon-qw2tv\" (UID: \"0480eb75-3ca1-46ba-9606-ec9e98dc9c4c\") " pod="openshift-multus/network-metrics-daemon-qw2tv"
Aug 04 18:00:52.794381 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:52.794364    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Aug 04 18:00:52.794451 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:52.794432    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Aug 04 18:00:52.803901 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:52.803876    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/31fd88d9-8ce3-4056-a185-4427ec8939c1-original-pull-secret\") pod \"global-pull-secret-syncer-lqxxl\" (UID: \"31fd88d9-8ce3-4056-a185-4427ec8939c1\") " pod="kube-system/global-pull-secret-syncer-lqxxl"
Aug 04 18:00:52.803977 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:52.803920    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0480eb75-3ca1-46ba-9606-ec9e98dc9c4c-metrics-certs\") pod \"network-metrics-daemon-qw2tv\" (UID: \"0480eb75-3ca1-46ba-9606-ec9e98dc9c4c\") " pod="openshift-multus/network-metrics-daemon-qw2tv"
Aug 04 18:00:52.847084 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:52.847062    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lqxxl"
Aug 04 18:00:52.857631 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:52.857610    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-88fc8\""
Aug 04 18:00:52.864683 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:52.864657    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-qxtnm\""
Aug 04 18:00:52.864763 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:52.864662    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qw2tv"
Aug 04 18:00:52.872072 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:52.872053    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jxnpj"
Aug 04 18:00:52.940591 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:52.940122    2583 scope.go:117] "RemoveContainer" containerID="1d5a2cb734fa3629814b2e3d61648e52018126be65deed695e63a8b5505d2f25"
Aug 04 18:00:52.993974 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:52.993947    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-lqxxl"]
Aug 04 18:00:52.996775 ip-10-0-135-124 kubenswrapper[2583]: W0804 18:00:52.996752    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31fd88d9_8ce3_4056_a185_4427ec8939c1.slice/crio-6c1deb7549a5d25bc11a38b53490f99729fda0fd276412ea401346558b7b77a0 WatchSource:0}: Error finding container 6c1deb7549a5d25bc11a38b53490f99729fda0fd276412ea401346558b7b77a0: Status 404 returned error can't find the container with id 6c1deb7549a5d25bc11a38b53490f99729fda0fd276412ea401346558b7b77a0
Aug 04 18:00:53.022569 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:53.022542    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-qw2tv"]
Aug 04 18:00:53.023215 ip-10-0-135-124 kubenswrapper[2583]: W0804 18:00:53.023192    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0480eb75_3ca1_46ba_9606_ec9e98dc9c4c.slice/crio-9e4f71571f329e0ac50c9930d50e6e36c7817b8bc7c9a4c6b8a812292a5b7816 WatchSource:0}: Error finding container 9e4f71571f329e0ac50c9930d50e6e36c7817b8bc7c9a4c6b8a812292a5b7816: Status 404 returned error can't find the container with id 9e4f71571f329e0ac50c9930d50e6e36c7817b8bc7c9a4c6b8a812292a5b7816
Aug 04 18:00:53.033087 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:53.033061    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-jxnpj"]
Aug 04 18:00:53.035604 ip-10-0-135-124 kubenswrapper[2583]: W0804 18:00:53.035580    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43697de6_c2bd_48c6_81eb_1f37fa5d3174.slice/crio-0b571d10f820460eb44d90e8524141b6f36f3d6d412b53d846b7bdbe2dd592ad WatchSource:0}: Error finding container 0b571d10f820460eb44d90e8524141b6f36f3d6d412b53d846b7bdbe2dd592ad: Status 404 returned error can't find the container with id 0b571d10f820460eb44d90e8524141b6f36f3d6d412b53d846b7bdbe2dd592ad
Aug 04 18:00:53.424002 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:53.423914    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-lqxxl" event={"ID":"31fd88d9-8ce3-4056-a185-4427ec8939c1","Type":"ContainerStarted","Data":"6c1deb7549a5d25bc11a38b53490f99729fda0fd276412ea401346558b7b77a0"}
Aug 04 18:00:53.425811 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:53.425787    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-wskfb_6157c2d3-2fd0-4544-88ad-434ac75f8c04/console-operator/1.log"
Aug 04 18:00:53.425938 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:53.425888    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-5ccd6b445f-wskfb" event={"ID":"6157c2d3-2fd0-4544-88ad-434ac75f8c04","Type":"ContainerStarted","Data":"4d2407b01fb4f0ffb1032df3e155e13518cf2b107e4f96dd42e8542883e79f9d"}
Aug 04 18:00:53.426180 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:53.426157    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-5ccd6b445f-wskfb"
Aug 04 18:00:53.427464 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:53.427440    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-jxnpj" event={"ID":"43697de6-c2bd-48c6-81eb-1f37fa5d3174","Type":"ContainerStarted","Data":"347a7198939f9129b85d3646499a9d7cbdfd399332c09155bd16bb0ca1fd9238"}
Aug 04 18:00:53.427584 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:53.427501    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-jxnpj" event={"ID":"43697de6-c2bd-48c6-81eb-1f37fa5d3174","Type":"ContainerStarted","Data":"0b571d10f820460eb44d90e8524141b6f36f3d6d412b53d846b7bdbe2dd592ad"}
Aug 04 18:00:53.427584 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:53.427527    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-jxnpj"
Aug 04 18:00:53.428677 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:53.428655    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-qw2tv" event={"ID":"0480eb75-3ca1-46ba-9606-ec9e98dc9c4c","Type":"ContainerStarted","Data":"9e4f71571f329e0ac50c9930d50e6e36c7817b8bc7c9a4c6b8a812292a5b7816"}
Aug 04 18:00:53.445229 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:53.445184    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-5ccd6b445f-wskfb" podStartSLOduration=34.789796192 podStartE2EDuration="42.445168542s" totalImagesPullingTime="7.65537235s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:00:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:00:53.445104093 +0000 UTC m=+67.026322670" watchObservedRunningTime="2026-08-04 18:00:53.445168542 +0000 UTC m=+67.026387121"
Aug 04 18:00:53.464892 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:53.464855    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-jxnpj" podStartSLOduration=66.464843994 podStartE2EDuration="1m6.464843994s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 17:59:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:00:53.463964423 +0000 UTC m=+67.045183001" watchObservedRunningTime="2026-08-04 18:00:53.464843994 +0000 UTC m=+67.046062572"
Aug 04 18:00:53.698838 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:53.698804    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/88e72b20-e499-46f0-9374-258bf49f39c2-service-ca-bundle\") pod \"router-default-85c4d8f7fd-bm9rl\" (UID: \"88e72b20-e499-46f0-9374-258bf49f39c2\") " pod="openshift-ingress/router-default-85c4d8f7fd-bm9rl"
Aug 04 18:00:53.698996 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:53.698902    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/88e72b20-e499-46f0-9374-258bf49f39c2-metrics-certs\") pod \"router-default-85c4d8f7fd-bm9rl\" (UID: \"88e72b20-e499-46f0-9374-258bf49f39c2\") " pod="openshift-ingress/router-default-85c4d8f7fd-bm9rl"
Aug 04 18:00:53.698996 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:53.698923    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4704d0e3-b459-422d-92d9-0e9ec9bbe24f-registry-tls\") pod \"image-registry-74cb995cd-pjjzg\" (UID: \"4704d0e3-b459-422d-92d9-0e9ec9bbe24f\") " pod="openshift-image-registry/image-registry-74cb995cd-pjjzg"
Aug 04 18:00:53.698996 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:53.698942    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3fb60b38-9cb9-4a07-9334-1b914ddd4a62-metrics-tls\") pod \"dns-default-7wnlf\" (UID: \"3fb60b38-9cb9-4a07-9334-1b914ddd4a62\") " pod="openshift-dns/dns-default-7wnlf"
Aug 04 18:00:53.698996 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:53.698971    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7ef28a34-7ec8-49f8-a58c-920661139d9c-samples-operator-tls\") pod \"cluster-samples-operator-559cbf5745-qqg4c\" (UID: \"7ef28a34-7ec8-49f8-a58c-920661139d9c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-qqg4c"
Aug 04 18:00:53.699716 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:53.699687    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e079c95a-25bd-44e2-8f30-44d6cba4add7-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6f55c4f58d-fzk78\" (UID: \"e079c95a-25bd-44e2-8f30-44d6cba4add7\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-fzk78"
Aug 04 18:00:53.699829 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:53.699800    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/88e72b20-e499-46f0-9374-258bf49f39c2-service-ca-bundle\") pod \"router-default-85c4d8f7fd-bm9rl\" (UID: \"88e72b20-e499-46f0-9374-258bf49f39c2\") " pod="openshift-ingress/router-default-85c4d8f7fd-bm9rl"
Aug 04 18:00:53.704091 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:53.704059    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e079c95a-25bd-44e2-8f30-44d6cba4add7-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6f55c4f58d-fzk78\" (UID: \"e079c95a-25bd-44e2-8f30-44d6cba4add7\") " pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-fzk78"
Aug 04 18:00:53.709729 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:53.707997    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/88e72b20-e499-46f0-9374-258bf49f39c2-metrics-certs\") pod \"router-default-85c4d8f7fd-bm9rl\" (UID: \"88e72b20-e499-46f0-9374-258bf49f39c2\") " pod="openshift-ingress/router-default-85c4d8f7fd-bm9rl"
Aug 04 18:00:53.709729 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:53.707945    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4704d0e3-b459-422d-92d9-0e9ec9bbe24f-registry-tls\") pod \"image-registry-74cb995cd-pjjzg\" (UID: \"4704d0e3-b459-422d-92d9-0e9ec9bbe24f\") " pod="openshift-image-registry/image-registry-74cb995cd-pjjzg"
Aug 04 18:00:53.709729 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:53.708347    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7ef28a34-7ec8-49f8-a58c-920661139d9c-samples-operator-tls\") pod \"cluster-samples-operator-559cbf5745-qqg4c\" (UID: \"7ef28a34-7ec8-49f8-a58c-920661139d9c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-qqg4c"
Aug 04 18:00:53.709729 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:53.708743    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3fb60b38-9cb9-4a07-9334-1b914ddd4a62-metrics-tls\") pod \"dns-default-7wnlf\" (UID: \"3fb60b38-9cb9-4a07-9334-1b914ddd4a62\") " pod="openshift-dns/dns-default-7wnlf"
Aug 04 18:00:53.866044 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:53.866010    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-68qht\""
Aug 04 18:00:53.873670 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:53.873619    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-74cb995cd-pjjzg"
Aug 04 18:00:53.902435 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:53.902414    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-lq4sk\""
Aug 04 18:00:53.909891 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:53.909866    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-85c4d8f7fd-bm9rl"
Aug 04 18:00:53.919459 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:53.919438    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-csqgv\""
Aug 04 18:00:53.926965 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:53.926944    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-fzk78"
Aug 04 18:00:53.951512 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:53.951445    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-vbw88\""
Aug 04 18:00:53.957814 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:53.957788    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-qqg4c"
Aug 04 18:00:53.959326 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:53.959289    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-snn7x\""
Aug 04 18:00:53.965934 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:53.965912    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-7wnlf"
Aug 04 18:00:54.163754 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:54.163720    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-5ccd6b445f-wskfb"
Aug 04 18:00:54.308280 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:54.308250    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-qqg4c"]
Aug 04 18:00:54.436323 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:54.436245    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-qw2tv" event={"ID":"0480eb75-3ca1-46ba-9606-ec9e98dc9c4c","Type":"ContainerStarted","Data":"7ec2fc9b0c82e4011ee49e0ee972cf2929ae9450d8296f53238ef4bfe73f2e01"}
Aug 04 18:00:54.438861 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:54.438371    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-qqg4c" event={"ID":"7ef28a34-7ec8-49f8-a58c-920661139d9c","Type":"ContainerStarted","Data":"b8d1206abebbb5dfb1cde35be5ed199dccd27bb1ca1ccb62cff4ff11c73cabe6"}
Aug 04 18:00:54.537810 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:54.537778    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-fzk78"]
Aug 04 18:00:54.540242 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:54.539867    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-7wnlf"]
Aug 04 18:00:54.541492 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:54.541309    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-74cb995cd-pjjzg"]
Aug 04 18:00:54.542756 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:54.542736    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-85c4d8f7fd-bm9rl"]
Aug 04 18:00:54.543143 ip-10-0-135-124 kubenswrapper[2583]: W0804 18:00:54.543117    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode079c95a_25bd_44e2_8f30_44d6cba4add7.slice/crio-b07128b01418e8e4ab3efda4c36e0f90433caef7a1e99e6dd41cb0d80bf49640 WatchSource:0}: Error finding container b07128b01418e8e4ab3efda4c36e0f90433caef7a1e99e6dd41cb0d80bf49640: Status 404 returned error can't find the container with id b07128b01418e8e4ab3efda4c36e0f90433caef7a1e99e6dd41cb0d80bf49640
Aug 04 18:00:54.545456 ip-10-0-135-124 kubenswrapper[2583]: W0804 18:00:54.545430    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4704d0e3_b459_422d_92d9_0e9ec9bbe24f.slice/crio-28992e5e459c17cb3056ac9b5f022415e1520fb461e65a049a850432fad8a76b WatchSource:0}: Error finding container 28992e5e459c17cb3056ac9b5f022415e1520fb461e65a049a850432fad8a76b: Status 404 returned error can't find the container with id 28992e5e459c17cb3056ac9b5f022415e1520fb461e65a049a850432fad8a76b
Aug 04 18:00:54.546869 ip-10-0-135-124 kubenswrapper[2583]: W0804 18:00:54.546845    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fb60b38_9cb9_4a07_9334_1b914ddd4a62.slice/crio-3a1ffd68c3cb3937eb041431755610c7233a26bd00de71cab4fbb504b3572a29 WatchSource:0}: Error finding container 3a1ffd68c3cb3937eb041431755610c7233a26bd00de71cab4fbb504b3572a29: Status 404 returned error can't find the container with id 3a1ffd68c3cb3937eb041431755610c7233a26bd00de71cab4fbb504b3572a29
Aug 04 18:00:54.547547 ip-10-0-135-124 kubenswrapper[2583]: W0804 18:00:54.547525    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88e72b20_e499_46f0_9374_258bf49f39c2.slice/crio-573ea58c4f540c2b6d6eef733e83b9980fe4937898f9eca10903151f37ac6409 WatchSource:0}: Error finding container 573ea58c4f540c2b6d6eef733e83b9980fe4937898f9eca10903151f37ac6409: Status 404 returned error can't find the container with id 573ea58c4f540c2b6d6eef733e83b9980fe4937898f9eca10903151f37ac6409
Aug 04 18:00:55.444115 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:55.444076    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-fzk78" event={"ID":"e079c95a-25bd-44e2-8f30-44d6cba4add7","Type":"ContainerStarted","Data":"b07128b01418e8e4ab3efda4c36e0f90433caef7a1e99e6dd41cb0d80bf49640"}
Aug 04 18:00:55.446234 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:55.446199    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-85c4d8f7fd-bm9rl" event={"ID":"88e72b20-e499-46f0-9374-258bf49f39c2","Type":"ContainerStarted","Data":"63d0914538ae47950c6fdbd80bd8bf2f536c4fe8d9ba7d921f9403841b64a92f"}
Aug 04 18:00:55.446370 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:55.446254    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-85c4d8f7fd-bm9rl" event={"ID":"88e72b20-e499-46f0-9374-258bf49f39c2","Type":"ContainerStarted","Data":"573ea58c4f540c2b6d6eef733e83b9980fe4937898f9eca10903151f37ac6409"}
Aug 04 18:00:55.447935 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:55.447908    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-7wnlf" event={"ID":"3fb60b38-9cb9-4a07-9334-1b914ddd4a62","Type":"ContainerStarted","Data":"3a1ffd68c3cb3937eb041431755610c7233a26bd00de71cab4fbb504b3572a29"}
Aug 04 18:00:55.450664 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:55.450273    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-qw2tv" event={"ID":"0480eb75-3ca1-46ba-9606-ec9e98dc9c4c","Type":"ContainerStarted","Data":"c8de6ccb5f00c38da537aa63e9c83b9e50e59f54a85e2dfd95d0a21fb0807bd3"}
Aug 04 18:00:55.452255 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:55.452207    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-74cb995cd-pjjzg" event={"ID":"4704d0e3-b459-422d-92d9-0e9ec9bbe24f","Type":"ContainerStarted","Data":"dd8afcc093487cef90954182a702c1da0525a446e28c40cfcef0289b87ae9b88"}
Aug 04 18:00:55.452255 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:55.452249    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-74cb995cd-pjjzg" event={"ID":"4704d0e3-b459-422d-92d9-0e9ec9bbe24f","Type":"ContainerStarted","Data":"28992e5e459c17cb3056ac9b5f022415e1520fb461e65a049a850432fad8a76b"}
Aug 04 18:00:55.452431 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:55.452401    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-74cb995cd-pjjzg"
Aug 04 18:00:55.468855 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:55.468695    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-85c4d8f7fd-bm9rl" podStartSLOduration=44.468675205 podStartE2EDuration="44.468675205s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:00:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:00:55.467586315 +0000 UTC m=+69.048804892" watchObservedRunningTime="2026-08-04 18:00:55.468675205 +0000 UTC m=+69.049893787"
Aug 04 18:00:55.491495 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:55.491425    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-74cb995cd-pjjzg" podStartSLOduration=67.491412949 podStartE2EDuration="1m7.491412949s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 17:59:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:00:55.489700593 +0000 UTC m=+69.070919172" watchObservedRunningTime="2026-08-04 18:00:55.491412949 +0000 UTC m=+69.072631528"
Aug 04 18:00:55.509210 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:55.508751    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-qw2tv" podStartSLOduration=67.397091385 podStartE2EDuration="1m8.508735104s" totalImagesPullingTime="1.111643719s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 17:59:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:00:55.508368879 +0000 UTC m=+69.089587458" watchObservedRunningTime="2026-08-04 18:00:55.508735104 +0000 UTC m=+69.089953684"
Aug 04 18:00:55.911267 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:55.911112    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-85c4d8f7fd-bm9rl"
Aug 04 18:00:55.914405 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:55.914157    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-85c4d8f7fd-bm9rl"
Aug 04 18:00:56.019641 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:56.019608    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6d05efcb-b6fb-485c-b472-bd406b8e729f-networking-console-plugin-cert\") pod \"networking-console-plugin-f77f48c6d-lt6n7\" (UID: \"6d05efcb-b6fb-485c-b472-bd406b8e729f\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-lt6n7"
Aug 04 18:00:56.028041 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:56.028007    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6d05efcb-b6fb-485c-b472-bd406b8e729f-networking-console-plugin-cert\") pod \"networking-console-plugin-f77f48c6d-lt6n7\" (UID: \"6d05efcb-b6fb-485c-b472-bd406b8e729f\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-lt6n7"
Aug 04 18:00:56.255021 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:56.254960    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-865jq\""
Aug 04 18:00:56.255224 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:56.255192    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-f77f48c6d-lt6n7"
Aug 04 18:00:56.472369 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:56.472314    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-85c4d8f7fd-bm9rl"
Aug 04 18:00:56.473530 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:56.473494    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-85c4d8f7fd-bm9rl"
Aug 04 18:00:57.120933 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:57.120899    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-77c974ffcf-7whsw"]
Aug 04 18:00:57.138408 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:57.138373    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-77c974ffcf-7whsw"]
Aug 04 18:00:57.138556 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:57.138515    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-77c974ffcf-7whsw"
Aug 04 18:00:57.141617 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:57.141593    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Aug 04 18:00:57.141722 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:57.141621    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Aug 04 18:00:57.141896 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:57.141879    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-t8f7d\""
Aug 04 18:00:57.231133 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:57.231099    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9glzt\" (UniqueName: \"kubernetes.io/projected/35f07db1-b2dd-45a1-a0d1-eec05bff872f-kube-api-access-9glzt\") pod \"downloads-77c974ffcf-7whsw\" (UID: \"35f07db1-b2dd-45a1-a0d1-eec05bff872f\") " pod="openshift-console/downloads-77c974ffcf-7whsw"
Aug 04 18:00:57.331761 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:57.331694    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9glzt\" (UniqueName: \"kubernetes.io/projected/35f07db1-b2dd-45a1-a0d1-eec05bff872f-kube-api-access-9glzt\") pod \"downloads-77c974ffcf-7whsw\" (UID: \"35f07db1-b2dd-45a1-a0d1-eec05bff872f\") " pod="openshift-console/downloads-77c974ffcf-7whsw"
Aug 04 18:00:57.339657 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:57.339638    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9glzt\" (UniqueName: \"kubernetes.io/projected/35f07db1-b2dd-45a1-a0d1-eec05bff872f-kube-api-access-9glzt\") pod \"downloads-77c974ffcf-7whsw\" (UID: \"35f07db1-b2dd-45a1-a0d1-eec05bff872f\") " pod="openshift-console/downloads-77c974ffcf-7whsw"
Aug 04 18:00:57.451298 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:00:57.451272    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-77c974ffcf-7whsw"
Aug 04 18:01:00.065192 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:01:00.065087    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-f77f48c6d-lt6n7"]
Aug 04 18:01:00.070822 ip-10-0-135-124 kubenswrapper[2583]: W0804 18:01:00.070790    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d05efcb_b6fb_485c_b472_bd406b8e729f.slice/crio-6602c45014c6a9c2b0c856c251a45bf6e6949a4bfbb0d12acfefda532f20b18d WatchSource:0}: Error finding container 6602c45014c6a9c2b0c856c251a45bf6e6949a4bfbb0d12acfefda532f20b18d: Status 404 returned error can't find the container with id 6602c45014c6a9c2b0c856c251a45bf6e6949a4bfbb0d12acfefda532f20b18d
Aug 04 18:01:00.090698 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:01:00.090654    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-77c974ffcf-7whsw"]
Aug 04 18:01:00.093193 ip-10-0-135-124 kubenswrapper[2583]: W0804 18:01:00.093163    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35f07db1_b2dd_45a1_a0d1_eec05bff872f.slice/crio-4b23e37ed0482361404e118399ef1f5f8e4ed6320279f58bbcc981beed0d7007 WatchSource:0}: Error finding container 4b23e37ed0482361404e118399ef1f5f8e4ed6320279f58bbcc981beed0d7007: Status 404 returned error can't find the container with id 4b23e37ed0482361404e118399ef1f5f8e4ed6320279f58bbcc981beed0d7007
Aug 04 18:01:00.490060 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:01:00.489981    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-fzk78" event={"ID":"e079c95a-25bd-44e2-8f30-44d6cba4add7","Type":"ContainerStarted","Data":"fedb2d26809590d8a08a8b86392f566b66e64c392b4eb84cab7a2f89adae2c70"}
Aug 04 18:01:00.492258 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:01:00.492230    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-qqg4c" event={"ID":"7ef28a34-7ec8-49f8-a58c-920661139d9c","Type":"ContainerStarted","Data":"18690c22ef02ccc5538ee2dd9d03c9178d4980d88909c40172c7b382ed72ad43"}
Aug 04 18:01:00.492400 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:01:00.492288    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-qqg4c" event={"ID":"7ef28a34-7ec8-49f8-a58c-920661139d9c","Type":"ContainerStarted","Data":"a0e543a5cbde897aa85efa3351ba08b28a60e7106e7090eeaa93bc021e1fa068"}
Aug 04 18:01:00.494652 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:01:00.494625    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-7wnlf" event={"ID":"3fb60b38-9cb9-4a07-9334-1b914ddd4a62","Type":"ContainerStarted","Data":"a08d54785513bce798918a59171c23a5351cf1b7b66b51ef0f97b2d92e7e32ce"}
Aug 04 18:01:00.494760 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:01:00.494683    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-7wnlf"
Aug 04 18:01:00.494760 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:01:00.494708    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-7wnlf" event={"ID":"3fb60b38-9cb9-4a07-9334-1b914ddd4a62","Type":"ContainerStarted","Data":"45136cf1b4e01f49e64e311d7d35c9dbe63292de7573d0cb7ea00f27a667c853"}
Aug 04 18:01:00.495997 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:01:00.495963    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-77c974ffcf-7whsw" event={"ID":"35f07db1-b2dd-45a1-a0d1-eec05bff872f","Type":"ContainerStarted","Data":"4b23e37ed0482361404e118399ef1f5f8e4ed6320279f58bbcc981beed0d7007"}
Aug 04 18:01:00.497303 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:01:00.497278    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-f77f48c6d-lt6n7" event={"ID":"6d05efcb-b6fb-485c-b472-bd406b8e729f","Type":"ContainerStarted","Data":"6602c45014c6a9c2b0c856c251a45bf6e6949a4bfbb0d12acfefda532f20b18d"}
Aug 04 18:01:00.498875 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:01:00.498856    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-lqxxl" event={"ID":"31fd88d9-8ce3-4056-a185-4427ec8939c1","Type":"ContainerStarted","Data":"462b7d96aad71b0b3ba8a10f7bd7056a8ce533e4ba95dc663fe87e7b8719116c"}
Aug 04 18:01:00.515552 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:01:00.515514    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-6f55c4f58d-fzk78" podStartSLOduration=44.148323745 podStartE2EDuration="49.51550171s" totalImagesPullingTime="5.367177965s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:00:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:01:00.514363673 +0000 UTC m=+74.095582252" watchObservedRunningTime="2026-08-04 18:01:00.51550171 +0000 UTC m=+74.096720278"
Aug 04 18:01:00.533152 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:01:00.533114    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-7wnlf" podStartSLOduration=34.169259888 podStartE2EDuration="39.533102573s" totalImagesPullingTime="5.363842685s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:00:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:01:00.532209231 +0000 UTC m=+74.113427809" watchObservedRunningTime="2026-08-04 18:01:00.533102573 +0000 UTC m=+74.114321183"
Aug 04 18:01:00.554374 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:01:00.554336    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-lqxxl" podStartSLOduration=66.631117335 podStartE2EDuration="1m13.554324153s" totalImagesPullingTime="6.923206818s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 17:59:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:01:00.553813029 +0000 UTC m=+74.135031607" watchObservedRunningTime="2026-08-04 18:01:00.554324153 +0000 UTC m=+74.135542709"
Aug 04 18:01:00.571230 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:01:00.571194    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-559cbf5745-qqg4c" podStartSLOduration=44.014141166 podStartE2EDuration="49.571181891s" totalImagesPullingTime="5.557040725s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:00:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:01:00.57082523 +0000 UTC m=+74.152043853" watchObservedRunningTime="2026-08-04 18:01:00.571181891 +0000 UTC m=+74.152400492"
Aug 04 18:01:02.506819 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:01:02.506782    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-f77f48c6d-lt6n7" event={"ID":"6d05efcb-b6fb-485c-b472-bd406b8e729f","Type":"ContainerStarted","Data":"b18b7f58dafc37d40c9413418880e0443af5d10c21154f262d38eea1c39e32f9"}
Aug 04 18:01:02.527485 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:01:02.527422    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-f77f48c6d-lt6n7" podStartSLOduration=36.855110979 podStartE2EDuration="38.527405079s" totalImagesPullingTime="1.6722941s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:00:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:01:02.526196031 +0000 UTC m=+76.107414608" watchObservedRunningTime="2026-08-04 18:01:02.527405079 +0000 UTC m=+76.108623651"
Aug 04 18:01:10.505305 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:01:10.505189    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-7wnlf"
Aug 04 18:01:16.067424 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:01:16.067384    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-9pl9x"]
Aug 04 18:01:16.072827 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:01:16.072787    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-9pl9x"
Aug 04 18:01:16.075644 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:01:16.075552    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Aug 04 18:01:16.075781 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:01:16.075698    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Aug 04 18:01:16.075781 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:01:16.075722    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Aug 04 18:01:16.075781 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:01:16.075708    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Aug 04 18:01:16.076770 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:01:16.076730    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-2sv5j\""
Aug 04 18:01:16.187679 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:01:16.187648    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/50eba630-4be0-4bfe-aa2f-36a5793d8f97-node-exporter-tls\") pod \"node-exporter-9pl9x\" (UID: \"50eba630-4be0-4bfe-aa2f-36a5793d8f97\") " pod="openshift-monitoring/node-exporter-9pl9x"
Aug 04 18:01:16.187857 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:01:16.187689    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/50eba630-4be0-4bfe-aa2f-36a5793d8f97-metrics-client-ca\") pod \"node-exporter-9pl9x\" (UID: \"50eba630-4be0-4bfe-aa2f-36a5793d8f97\") " pod="openshift-monitoring/node-exporter-9pl9x"
Aug 04 18:01:16.187857 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:01:16.187709    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9sb4z\" (UniqueName: \"kubernetes.io/projected/50eba630-4be0-4bfe-aa2f-36a5793d8f97-kube-api-access-9sb4z\") pod \"node-exporter-9pl9x\" (UID: \"50eba630-4be0-4bfe-aa2f-36a5793d8f97\") " pod="openshift-monitoring/node-exporter-9pl9x"
Aug 04 18:01:16.187857 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:01:16.187733    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/50eba630-4be0-4bfe-aa2f-36a5793d8f97-node-exporter-textfile\") pod \"node-exporter-9pl9x\" (UID: \"50eba630-4be0-4bfe-aa2f-36a5793d8f97\") " pod="openshift-monitoring/node-exporter-9pl9x"
Aug 04 18:01:16.187857 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:01:16.187792    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/50eba630-4be0-4bfe-aa2f-36a5793d8f97-root\") pod \"node-exporter-9pl9x\" (UID: \"50eba630-4be0-4bfe-aa2f-36a5793d8f97\") " pod="openshift-monitoring/node-exporter-9pl9x"
Aug 04 18:01:16.188076 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:01:16.187870    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/50eba630-4be0-4bfe-aa2f-36a5793d8f97-node-exporter-accelerators-collector-config\") pod \"node-exporter-9pl9x\" (UID: \"50eba630-4be0-4bfe-aa2f-36a5793d8f97\") " pod="openshift-monitoring/node-exporter-9pl9x"
Aug 04 18:01:16.188076 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:01:16.187908    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/50eba630-4be0-4bfe-aa2f-36a5793d8f97-sys\") pod \"node-exporter-9pl9x\" (UID: \"50eba630-4be0-4bfe-aa2f-36a5793d8f97\") " pod="openshift-monitoring/node-exporter-9pl9x"
Aug 04 18:01:16.188076 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:01:16.187925    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/50eba630-4be0-4bfe-aa2f-36a5793d8f97-node-exporter-wtmp\") pod \"node-exporter-9pl9x\" (UID: \"50eba630-4be0-4bfe-aa2f-36a5793d8f97\") " pod="openshift-monitoring/node-exporter-9pl9x"
Aug 04 18:01:16.188076 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:01:16.188002    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/50eba630-4be0-4bfe-aa2f-36a5793d8f97-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9pl9x\" (UID: \"50eba630-4be0-4bfe-aa2f-36a5793d8f97\") " pod="openshift-monitoring/node-exporter-9pl9x"
Aug 04 18:01:16.288913 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:01:16.288878    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/50eba630-4be0-4bfe-aa2f-36a5793d8f97-metrics-client-ca\") pod \"node-exporter-9pl9x\" (UID: \"50eba630-4be0-4bfe-aa2f-36a5793d8f97\") " pod="openshift-monitoring/node-exporter-9pl9x"
Aug 04 18:01:16.289087 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:01:16.288924    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9sb4z\" (UniqueName: \"kubernetes.io/projected/50eba630-4be0-4bfe-aa2f-36a5793d8f97-kube-api-access-9sb4z\") pod \"node-exporter-9pl9x\" (UID: \"50eba630-4be0-4bfe-aa2f-36a5793d8f97\") " pod="openshift-monitoring/node-exporter-9pl9x"
Aug 04 18:01:16.289087 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:01:16.288951    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/50eba630-4be0-4bfe-aa2f-36a5793d8f97-node-exporter-textfile\") pod \"node-exporter-9pl9x\" (UID: \"50eba630-4be0-4bfe-aa2f-36a5793d8f97\") " pod="openshift-monitoring/node-exporter-9pl9x"
Aug 04 18:01:16.289087 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:01:16.288980    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/50eba630-4be0-4bfe-aa2f-36a5793d8f97-root\") pod \"node-exporter-9pl9x\" (UID: \"50eba630-4be0-4bfe-aa2f-36a5793d8f97\") " pod="openshift-monitoring/node-exporter-9pl9x"
Aug 04 18:01:16.289087 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:01:16.289028    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/50eba630-4be0-4bfe-aa2f-36a5793d8f97-node-exporter-accelerators-collector-config\") pod \"node-exporter-9pl9x\" (UID: \"50eba630-4be0-4bfe-aa2f-36a5793d8f97\") " pod="openshift-monitoring/node-exporter-9pl9x"
Aug 04 18:01:16.289087 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:01:16.289059    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/50eba630-4be0-4bfe-aa2f-36a5793d8f97-sys\") pod \"node-exporter-9pl9x\" (UID: \"50eba630-4be0-4bfe-aa2f-36a5793d8f97\") " pod="openshift-monitoring/node-exporter-9pl9x"
Aug 04 18:01:16.289087 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:01:16.289080    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/50eba630-4be0-4bfe-aa2f-36a5793d8f97-node-exporter-wtmp\") pod \"node-exporter-9pl9x\" (UID: \"50eba630-4be0-4bfe-aa2f-36a5793d8f97\") " pod="openshift-monitoring/node-exporter-9pl9x"
Aug 04 18:01:16.289332 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:01:16.289117    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/50eba630-4be0-4bfe-aa2f-36a5793d8f97-root\") pod \"node-exporter-9pl9x\" (UID: \"50eba630-4be0-4bfe-aa2f-36a5793d8f97\") " pod="openshift-monitoring/node-exporter-9pl9x"
Aug 04 18:01:16.289332 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:01:16.289148    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/50eba630-4be0-4bfe-aa2f-36a5793d8f97-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9pl9x\" (UID: \"50eba630-4be0-4bfe-aa2f-36a5793d8f97\") " pod="openshift-monitoring/node-exporter-9pl9x"
Aug 04 18:01:16.289332 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:01:16.289177    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/50eba630-4be0-4bfe-aa2f-36a5793d8f97-node-exporter-tls\") pod \"node-exporter-9pl9x\" (UID: \"50eba630-4be0-4bfe-aa2f-36a5793d8f97\") " pod="openshift-monitoring/node-exporter-9pl9x"
Aug 04 18:01:16.289332 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:01:16.289186    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/50eba630-4be0-4bfe-aa2f-36a5793d8f97-sys\") pod \"node-exporter-9pl9x\" (UID: \"50eba630-4be0-4bfe-aa2f-36a5793d8f97\") " pod="openshift-monitoring/node-exporter-9pl9x"
Aug 04 18:01:16.289457 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:01:16.289344    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/50eba630-4be0-4bfe-aa2f-36a5793d8f97-node-exporter-wtmp\") pod \"node-exporter-9pl9x\" (UID: \"50eba630-4be0-4bfe-aa2f-36a5793d8f97\") " pod="openshift-monitoring/node-exporter-9pl9x"
Aug 04 18:01:16.289457 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:01:16.289363    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/50eba630-4be0-4bfe-aa2f-36a5793d8f97-node-exporter-textfile\") pod \"node-exporter-9pl9x\" (UID: \"50eba630-4be0-4bfe-aa2f-36a5793d8f97\") " pod="openshift-monitoring/node-exporter-9pl9x"
Aug 04 18:01:16.289619 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:01:16.289592    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/50eba630-4be0-4bfe-aa2f-36a5793d8f97-metrics-client-ca\") pod \"node-exporter-9pl9x\" (UID: \"50eba630-4be0-4bfe-aa2f-36a5793d8f97\") " pod="openshift-monitoring/node-exporter-9pl9x"
Aug 04 18:01:16.289800 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:01:16.289777    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/50eba630-4be0-4bfe-aa2f-36a5793d8f97-node-exporter-accelerators-collector-config\") pod \"node-exporter-9pl9x\" (UID: \"50eba630-4be0-4bfe-aa2f-36a5793d8f97\") " pod="openshift-monitoring/node-exporter-9pl9x"
Aug 04 18:01:16.291845 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:01:16.291822    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/50eba630-4be0-4bfe-aa2f-36a5793d8f97-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9pl9x\" (UID: \"50eba630-4be0-4bfe-aa2f-36a5793d8f97\") " pod="openshift-monitoring/node-exporter-9pl9x"
Aug 04 18:01:16.291925 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:01:16.291893    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/50eba630-4be0-4bfe-aa2f-36a5793d8f97-node-exporter-tls\") pod \"node-exporter-9pl9x\" (UID: \"50eba630-4be0-4bfe-aa2f-36a5793d8f97\") " pod="openshift-monitoring/node-exporter-9pl9x"
Aug 04 18:01:16.299796 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:01:16.299751    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9sb4z\" (UniqueName: \"kubernetes.io/projected/50eba630-4be0-4bfe-aa2f-36a5793d8f97-kube-api-access-9sb4z\") pod \"node-exporter-9pl9x\" (UID: \"50eba630-4be0-4bfe-aa2f-36a5793d8f97\") " pod="openshift-monitoring/node-exporter-9pl9x"
Aug 04 18:01:16.384453 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:01:16.384370    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-9pl9x"
Aug 04 18:01:16.476988 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:01:16.476957    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-74cb995cd-pjjzg"
Aug 04 18:01:18.768749 ip-10-0-135-124 kubenswrapper[2583]: W0804 18:01:18.768713    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50eba630_4be0_4bfe_aa2f_36a5793d8f97.slice/crio-c6c57e79936c45ad14ac0114a842effa3d66029f1e6a02f63d6d62e144f52240 WatchSource:0}: Error finding container c6c57e79936c45ad14ac0114a842effa3d66029f1e6a02f63d6d62e144f52240: Status 404 returned error can't find the container with id c6c57e79936c45ad14ac0114a842effa3d66029f1e6a02f63d6d62e144f52240
Aug 04 18:01:19.026997 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:01:19.026917    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-74cb995cd-pjjzg"]
Aug 04 18:01:19.590092 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:01:19.589944    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-77c974ffcf-7whsw" event={"ID":"35f07db1-b2dd-45a1-a0d1-eec05bff872f","Type":"ContainerStarted","Data":"2949fe332ffc23d2a7be06a63f733ba52c03cc0d9636182819e55a9a4d7885dd"}
Aug 04 18:01:19.590609 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:01:19.590546    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-77c974ffcf-7whsw"
Aug 04 18:01:19.592937 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:01:19.592900    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9pl9x" event={"ID":"50eba630-4be0-4bfe-aa2f-36a5793d8f97","Type":"ContainerStarted","Data":"c6c57e79936c45ad14ac0114a842effa3d66029f1e6a02f63d6d62e144f52240"}
Aug 04 18:01:19.606215 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:01:19.606184    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-77c974ffcf-7whsw"
Aug 04 18:01:19.626293 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:01:19.626243    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-77c974ffcf-7whsw" podStartSLOduration=3.8721944710000002 podStartE2EDuration="22.626223272s" totalImagesPullingTime="18.754028801s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:00:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:01:19.624263361 +0000 UTC m=+93.205481941" watchObservedRunningTime="2026-08-04 18:01:19.626223272 +0000 UTC m=+93.207441851"
Aug 04 18:01:20.599250 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:01:20.599180    2583 generic.go:356] "Generic (PLEG): container finished" podID="50eba630-4be0-4bfe-aa2f-36a5793d8f97" containerID="b7cc23b7b353869edbdd016f4be111b77f7d642b67640db74df87bb81c1bd152" exitCode=0
Aug 04 18:01:20.599646 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:01:20.599277    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9pl9x" event={"ID":"50eba630-4be0-4bfe-aa2f-36a5793d8f97","Type":"ContainerDied","Data":"b7cc23b7b353869edbdd016f4be111b77f7d642b67640db74df87bb81c1bd152"}
Aug 04 18:01:21.606668 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:01:21.606627    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9pl9x" event={"ID":"50eba630-4be0-4bfe-aa2f-36a5793d8f97","Type":"ContainerStarted","Data":"1a3d511fb3bd3a642d58c897c5739ea578c416bd27afa9066ba11e6b18e43c71"}
Aug 04 18:01:21.607217 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:01:21.606691    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9pl9x" event={"ID":"50eba630-4be0-4bfe-aa2f-36a5793d8f97","Type":"ContainerStarted","Data":"c1e86885ba1e06eb8624128f0b623b31daef2828dab2326a31d3d852ed675ce8"}
Aug 04 18:01:21.636397 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:01:21.636346    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-9pl9x" podStartSLOduration=3.964560894 podStartE2EDuration="5.636330954s" totalImagesPullingTime="1.578564055s" totalInitContainerRuntime="93.206005ms" isStatefulPod=true podCreationTimestamp="2026-08-04 18:01:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:01:21.633962441 +0000 UTC m=+95.215181020" watchObservedRunningTime="2026-08-04 18:01:21.636330954 +0000 UTC m=+95.217549547"
Aug 04 18:01:24.442109 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:01:24.442074    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-jxnpj"
Aug 04 18:01:27.565797 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:01:27.565760    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7ffcf99db4-k6vvm"]
Aug 04 18:01:27.610049 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:01:27.610020    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7ffcf99db4-k6vvm"]
Aug 04 18:01:27.610199 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:01:27.610125    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7ffcf99db4-k6vvm"
Aug 04 18:01:27.613727 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:01:27.613695    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-jw8gn\""
Aug 04 18:01:27.615881 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:01:27.615857    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Aug 04 18:01:27.615881 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:01:27.615875    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Aug 04 18:01:27.616092 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:01:27.615858    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Aug 04 18:01:27.616220 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:01:27.616201    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Aug 04 18:01:27.616376 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:01:27.616361    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Aug 04 18:01:27.622488 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:01:27.622447    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Aug 04 18:01:27.684564 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:01:27.684536    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/144865c4-61dd-4fa5-b40d-f08e404fdc9d-console-oauth-config\") pod \"console-7ffcf99db4-k6vvm\" (UID: \"144865c4-61dd-4fa5-b40d-f08e404fdc9d\") " pod="openshift-console/console-7ffcf99db4-k6vvm"
Aug 04 18:01:27.684564 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:01:27.684565    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/144865c4-61dd-4fa5-b40d-f08e404fdc9d-console-config\") pod \"console-7ffcf99db4-k6vvm\" (UID: \"144865c4-61dd-4fa5-b40d-f08e404fdc9d\") " pod="openshift-console/console-7ffcf99db4-k6vvm"
Aug 04 18:01:27.684762 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:01:27.684603    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/144865c4-61dd-4fa5-b40d-f08e404fdc9d-service-ca\") pod \"console-7ffcf99db4-k6vvm\" (UID: \"144865c4-61dd-4fa5-b40d-f08e404fdc9d\") " pod="openshift-console/console-7ffcf99db4-k6vvm"
Aug 04 18:01:27.684762 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:01:27.684624    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/144865c4-61dd-4fa5-b40d-f08e404fdc9d-trusted-ca-bundle\") pod \"console-7ffcf99db4-k6vvm\" (UID: \"144865c4-61dd-4fa5-b40d-f08e404fdc9d\") " pod="openshift-console/console-7ffcf99db4-k6vvm"
Aug 04 18:01:27.684762 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:01:27.684713    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7hpc\" (UniqueName: \"kubernetes.io/projected/144865c4-61dd-4fa5-b40d-f08e404fdc9d-kube-api-access-v7hpc\") pod \"console-7ffcf99db4-k6vvm\" (UID: \"144865c4-61dd-4fa5-b40d-f08e404fdc9d\") " pod="openshift-console/console-7ffcf99db4-k6vvm"
Aug 04 18:01:27.684887 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:01:27.684767    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/144865c4-61dd-4fa5-b40d-f08e404fdc9d-oauth-serving-cert\") pod \"console-7ffcf99db4-k6vvm\" (UID: \"144865c4-61dd-4fa5-b40d-f08e404fdc9d\") " pod="openshift-console/console-7ffcf99db4-k6vvm"
Aug 04 18:01:27.684887 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:01:27.684832    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/144865c4-61dd-4fa5-b40d-f08e404fdc9d-console-serving-cert\") pod \"console-7ffcf99db4-k6vvm\" (UID: \"144865c4-61dd-4fa5-b40d-f08e404fdc9d\") " pod="openshift-console/console-7ffcf99db4-k6vvm"
Aug 04 18:01:27.785658 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:01:27.785619    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/144865c4-61dd-4fa5-b40d-f08e404fdc9d-console-serving-cert\") pod \"console-7ffcf99db4-k6vvm\" (UID: \"144865c4-61dd-4fa5-b40d-f08e404fdc9d\") " pod="openshift-console/console-7ffcf99db4-k6vvm"
Aug 04 18:01:27.785821 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:01:27.785691    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/144865c4-61dd-4fa5-b40d-f08e404fdc9d-console-oauth-config\") pod \"console-7ffcf99db4-k6vvm\" (UID: \"144865c4-61dd-4fa5-b40d-f08e404fdc9d\") " pod="openshift-console/console-7ffcf99db4-k6vvm"
Aug 04 18:01:27.785821 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:01:27.785719    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/144865c4-61dd-4fa5-b40d-f08e404fdc9d-console-config\") pod \"console-7ffcf99db4-k6vvm\" (UID: \"144865c4-61dd-4fa5-b40d-f08e404fdc9d\") " pod="openshift-console/console-7ffcf99db4-k6vvm"
Aug 04 18:01:27.785821 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:01:27.785780    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/144865c4-61dd-4fa5-b40d-f08e404fdc9d-service-ca\") pod \"console-7ffcf99db4-k6vvm\" (UID: \"144865c4-61dd-4fa5-b40d-f08e404fdc9d\") " pod="openshift-console/console-7ffcf99db4-k6vvm"
Aug 04 18:01:27.786039 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:01:27.785839    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/144865c4-61dd-4fa5-b40d-f08e404fdc9d-trusted-ca-bundle\") pod \"console-7ffcf99db4-k6vvm\" (UID: \"144865c4-61dd-4fa5-b40d-f08e404fdc9d\") " pod="openshift-console/console-7ffcf99db4-k6vvm"
Aug 04 18:01:27.786039 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:01:27.785879    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7hpc\" (UniqueName: \"kubernetes.io/projected/144865c4-61dd-4fa5-b40d-f08e404fdc9d-kube-api-access-v7hpc\") pod \"console-7ffcf99db4-k6vvm\" (UID: \"144865c4-61dd-4fa5-b40d-f08e404fdc9d\") " pod="openshift-console/console-7ffcf99db4-k6vvm"
Aug 04 18:01:27.786039 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:01:27.785920    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/144865c4-61dd-4fa5-b40d-f08e404fdc9d-oauth-serving-cert\") pod \"console-7ffcf99db4-k6vvm\" (UID: \"144865c4-61dd-4fa5-b40d-f08e404fdc9d\") " pod="openshift-console/console-7ffcf99db4-k6vvm"
Aug 04 18:01:27.786584 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:01:27.786554    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/144865c4-61dd-4fa5-b40d-f08e404fdc9d-console-config\") pod \"console-7ffcf99db4-k6vvm\" (UID: \"144865c4-61dd-4fa5-b40d-f08e404fdc9d\") " pod="openshift-console/console-7ffcf99db4-k6vvm"
Aug 04 18:01:27.786708 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:01:27.786602    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/144865c4-61dd-4fa5-b40d-f08e404fdc9d-oauth-serving-cert\") pod \"console-7ffcf99db4-k6vvm\" (UID: \"144865c4-61dd-4fa5-b40d-f08e404fdc9d\") " pod="openshift-console/console-7ffcf99db4-k6vvm"
Aug 04 18:01:27.786708 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:01:27.786633    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/144865c4-61dd-4fa5-b40d-f08e404fdc9d-service-ca\") pod \"console-7ffcf99db4-k6vvm\" (UID: \"144865c4-61dd-4fa5-b40d-f08e404fdc9d\") " pod="openshift-console/console-7ffcf99db4-k6vvm"
Aug 04 18:01:27.786952 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:01:27.786931    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/144865c4-61dd-4fa5-b40d-f08e404fdc9d-trusted-ca-bundle\") pod \"console-7ffcf99db4-k6vvm\" (UID: \"144865c4-61dd-4fa5-b40d-f08e404fdc9d\") " pod="openshift-console/console-7ffcf99db4-k6vvm"
Aug 04 18:01:27.788295 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:01:27.788271    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/144865c4-61dd-4fa5-b40d-f08e404fdc9d-console-serving-cert\") pod \"console-7ffcf99db4-k6vvm\" (UID: \"144865c4-61dd-4fa5-b40d-f08e404fdc9d\") " pod="openshift-console/console-7ffcf99db4-k6vvm"
Aug 04 18:01:27.788295 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:01:27.788289    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/144865c4-61dd-4fa5-b40d-f08e404fdc9d-console-oauth-config\") pod \"console-7ffcf99db4-k6vvm\" (UID: \"144865c4-61dd-4fa5-b40d-f08e404fdc9d\") " pod="openshift-console/console-7ffcf99db4-k6vvm"
Aug 04 18:01:27.795484 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:01:27.795444    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7hpc\" (UniqueName: \"kubernetes.io/projected/144865c4-61dd-4fa5-b40d-f08e404fdc9d-kube-api-access-v7hpc\") pod \"console-7ffcf99db4-k6vvm\" (UID: \"144865c4-61dd-4fa5-b40d-f08e404fdc9d\") " pod="openshift-console/console-7ffcf99db4-k6vvm"
Aug 04 18:01:27.922131 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:01:27.922072    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7ffcf99db4-k6vvm"
Aug 04 18:01:28.069277 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:01:28.069251    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7ffcf99db4-k6vvm"]
Aug 04 18:01:28.071811 ip-10-0-135-124 kubenswrapper[2583]: W0804 18:01:28.071781    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod144865c4_61dd_4fa5_b40d_f08e404fdc9d.slice/crio-dcc43f27a54293014813e71c70029d7e69897668df65ffe1e83a0b23f582b630 WatchSource:0}: Error finding container dcc43f27a54293014813e71c70029d7e69897668df65ffe1e83a0b23f582b630: Status 404 returned error can't find the container with id dcc43f27a54293014813e71c70029d7e69897668df65ffe1e83a0b23f582b630
Aug 04 18:01:28.638247 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:01:28.638216    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7ffcf99db4-k6vvm" event={"ID":"144865c4-61dd-4fa5-b40d-f08e404fdc9d","Type":"ContainerStarted","Data":"dcc43f27a54293014813e71c70029d7e69897668df65ffe1e83a0b23f582b630"}
Aug 04 18:01:34.662717 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:01:34.662687    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7ffcf99db4-k6vvm" event={"ID":"144865c4-61dd-4fa5-b40d-f08e404fdc9d","Type":"ContainerStarted","Data":"d405ac3a7194f501d20d5c1aa762f04b164949287bbe36b0e31fd0e1a0f95bb4"}
Aug 04 18:01:34.682456 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:01:34.682409    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7ffcf99db4-k6vvm" podStartSLOduration=2.146467778 podStartE2EDuration="7.682393152s" totalImagesPullingTime="5.535925374s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:01:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:01:34.681086241 +0000 UTC m=+108.262304819" watchObservedRunningTime="2026-08-04 18:01:34.682393152 +0000 UTC m=+108.263611730"
Aug 04 18:01:37.922718 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:01:37.922677    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7ffcf99db4-k6vvm"
Aug 04 18:01:37.923130 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:01:37.922743    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7ffcf99db4-k6vvm"
Aug 04 18:01:37.927608 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:01:37.927588    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7ffcf99db4-k6vvm"
Aug 04 18:01:38.680816 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:01:38.680788    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7ffcf99db4-k6vvm"
Aug 04 18:01:42.693188 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:01:42.693153    2583 generic.go:356] "Generic (PLEG): container finished" podID="582f0d25-f53a-491b-ae73-d09ed1ab0501" containerID="a3b840c76b51b2faac32c58329c4188e07b82a92cccdec786b08d19af4e0444e" exitCode=0
Aug 04 18:01:42.693715 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:01:42.693225    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-6m4nc" event={"ID":"582f0d25-f53a-491b-ae73-d09ed1ab0501","Type":"ContainerDied","Data":"a3b840c76b51b2faac32c58329c4188e07b82a92cccdec786b08d19af4e0444e"}
Aug 04 18:01:42.693715 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:01:42.693617    2583 scope.go:117] "RemoveContainer" containerID="a3b840c76b51b2faac32c58329c4188e07b82a92cccdec786b08d19af4e0444e"
Aug 04 18:01:42.694720 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:01:42.694703    2583 generic.go:356] "Generic (PLEG): container finished" podID="24555f40-5be1-4a70-9e33-ba3c95c1e02f" containerID="735b60adf9b4da8976cbd6812dabcf77eaa1cdb45e4122579fe560862bad60b4" exitCode=0
Aug 04 18:01:42.694811 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:01:42.694736    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-77696685b-7mh27" event={"ID":"24555f40-5be1-4a70-9e33-ba3c95c1e02f","Type":"ContainerDied","Data":"735b60adf9b4da8976cbd6812dabcf77eaa1cdb45e4122579fe560862bad60b4"}
Aug 04 18:01:42.694993 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:01:42.694980    2583 scope.go:117] "RemoveContainer" containerID="735b60adf9b4da8976cbd6812dabcf77eaa1cdb45e4122579fe560862bad60b4"
Aug 04 18:01:43.701303 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:01:43.701265    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-77696685b-7mh27" event={"ID":"24555f40-5be1-4a70-9e33-ba3c95c1e02f","Type":"ContainerStarted","Data":"00e7770540af5635fae1f31ca68f6fe61b6b4e9d24ce6d1fefef05380f860ca6"}
Aug 04 18:01:43.703202 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:01:43.703172    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-6m4nc" event={"ID":"582f0d25-f53a-491b-ae73-d09ed1ab0501","Type":"ContainerStarted","Data":"41711f86e3225506694e3b3e649c68c12f212192a4dcfa18f2b0baf464effc5e"}
Aug 04 18:01:44.049831 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:01:44.049731    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-74cb995cd-pjjzg" podUID="4704d0e3-b459-422d-92d9-0e9ec9bbe24f" containerName="registry" containerID="cri-o://dd8afcc093487cef90954182a702c1da0525a446e28c40cfcef0289b87ae9b88" gracePeriod=30
Aug 04 18:01:44.309281 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:01:44.309221    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-74cb995cd-pjjzg"
Aug 04 18:01:44.418603 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:01:44.418577    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bg6t6\" (UniqueName: \"kubernetes.io/projected/4704d0e3-b459-422d-92d9-0e9ec9bbe24f-kube-api-access-bg6t6\") pod \"4704d0e3-b459-422d-92d9-0e9ec9bbe24f\" (UID: \"4704d0e3-b459-422d-92d9-0e9ec9bbe24f\") "
Aug 04 18:01:44.418781 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:01:44.418615    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4704d0e3-b459-422d-92d9-0e9ec9bbe24f-installation-pull-secrets\") pod \"4704d0e3-b459-422d-92d9-0e9ec9bbe24f\" (UID: \"4704d0e3-b459-422d-92d9-0e9ec9bbe24f\") "
Aug 04 18:01:44.418781 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:01:44.418648    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4704d0e3-b459-422d-92d9-0e9ec9bbe24f-trusted-ca\") pod \"4704d0e3-b459-422d-92d9-0e9ec9bbe24f\" (UID: \"4704d0e3-b459-422d-92d9-0e9ec9bbe24f\") "
Aug 04 18:01:44.418781 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:01:44.418667    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4704d0e3-b459-422d-92d9-0e9ec9bbe24f-bound-sa-token\") pod \"4704d0e3-b459-422d-92d9-0e9ec9bbe24f\" (UID: \"4704d0e3-b459-422d-92d9-0e9ec9bbe24f\") "
Aug 04 18:01:44.418781 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:01:44.418707    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4704d0e3-b459-422d-92d9-0e9ec9bbe24f-ca-trust-extracted\") pod \"4704d0e3-b459-422d-92d9-0e9ec9bbe24f\" (UID: \"4704d0e3-b459-422d-92d9-0e9ec9bbe24f\") "
Aug 04 18:01:44.418781 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:01:44.418738    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4704d0e3-b459-422d-92d9-0e9ec9bbe24f-registry-certificates\") pod \"4704d0e3-b459-422d-92d9-0e9ec9bbe24f\" (UID: \"4704d0e3-b459-422d-92d9-0e9ec9bbe24f\") "
Aug 04 18:01:44.418781 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:01:44.418760    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4704d0e3-b459-422d-92d9-0e9ec9bbe24f-registry-tls\") pod \"4704d0e3-b459-422d-92d9-0e9ec9bbe24f\" (UID: \"4704d0e3-b459-422d-92d9-0e9ec9bbe24f\") "
Aug 04 18:01:44.419095 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:01:44.418793    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4704d0e3-b459-422d-92d9-0e9ec9bbe24f-image-registry-private-configuration\") pod \"4704d0e3-b459-422d-92d9-0e9ec9bbe24f\" (UID: \"4704d0e3-b459-422d-92d9-0e9ec9bbe24f\") "
Aug 04 18:01:44.419234 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:01:44.419179    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4704d0e3-b459-422d-92d9-0e9ec9bbe24f-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "4704d0e3-b459-422d-92d9-0e9ec9bbe24f" (UID: "4704d0e3-b459-422d-92d9-0e9ec9bbe24f"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:01:44.419673 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:01:44.419637    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4704d0e3-b459-422d-92d9-0e9ec9bbe24f-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "4704d0e3-b459-422d-92d9-0e9ec9bbe24f" (UID: "4704d0e3-b459-422d-92d9-0e9ec9bbe24f"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:01:44.421264 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:01:44.421233    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4704d0e3-b459-422d-92d9-0e9ec9bbe24f-kube-api-access-bg6t6" (OuterVolumeSpecName: "kube-api-access-bg6t6") pod "4704d0e3-b459-422d-92d9-0e9ec9bbe24f" (UID: "4704d0e3-b459-422d-92d9-0e9ec9bbe24f"). InnerVolumeSpecName "kube-api-access-bg6t6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:01:44.421366 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:01:44.421335    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4704d0e3-b459-422d-92d9-0e9ec9bbe24f-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "4704d0e3-b459-422d-92d9-0e9ec9bbe24f" (UID: "4704d0e3-b459-422d-92d9-0e9ec9bbe24f"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:01:44.421428 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:01:44.421351    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4704d0e3-b459-422d-92d9-0e9ec9bbe24f-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "4704d0e3-b459-422d-92d9-0e9ec9bbe24f" (UID: "4704d0e3-b459-422d-92d9-0e9ec9bbe24f"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:01:44.421499 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:01:44.421422    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4704d0e3-b459-422d-92d9-0e9ec9bbe24f-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "4704d0e3-b459-422d-92d9-0e9ec9bbe24f" (UID: "4704d0e3-b459-422d-92d9-0e9ec9bbe24f"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:01:44.421542 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:01:44.421524    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4704d0e3-b459-422d-92d9-0e9ec9bbe24f-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "4704d0e3-b459-422d-92d9-0e9ec9bbe24f" (UID: "4704d0e3-b459-422d-92d9-0e9ec9bbe24f"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:01:44.427407 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:01:44.427384    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4704d0e3-b459-422d-92d9-0e9ec9bbe24f-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "4704d0e3-b459-422d-92d9-0e9ec9bbe24f" (UID: "4704d0e3-b459-422d-92d9-0e9ec9bbe24f"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:01:44.519673 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:01:44.519649    2583 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4704d0e3-b459-422d-92d9-0e9ec9bbe24f-registry-certificates\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:01:44.519673 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:01:44.519671    2583 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4704d0e3-b459-422d-92d9-0e9ec9bbe24f-registry-tls\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:01:44.519793 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:01:44.519681    2583 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4704d0e3-b459-422d-92d9-0e9ec9bbe24f-image-registry-private-configuration\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:01:44.519793 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:01:44.519690    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bg6t6\" (UniqueName: \"kubernetes.io/projected/4704d0e3-b459-422d-92d9-0e9ec9bbe24f-kube-api-access-bg6t6\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:01:44.519793 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:01:44.519699    2583 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4704d0e3-b459-422d-92d9-0e9ec9bbe24f-installation-pull-secrets\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:01:44.519793 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:01:44.519707    2583 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4704d0e3-b459-422d-92d9-0e9ec9bbe24f-trusted-ca\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:01:44.519793 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:01:44.519715    2583 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4704d0e3-b459-422d-92d9-0e9ec9bbe24f-bound-sa-token\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:01:44.519793 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:01:44.519722    2583 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4704d0e3-b459-422d-92d9-0e9ec9bbe24f-ca-trust-extracted\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:01:44.708155 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:01:44.708129    2583 generic.go:356] "Generic (PLEG): container finished" podID="4704d0e3-b459-422d-92d9-0e9ec9bbe24f" containerID="dd8afcc093487cef90954182a702c1da0525a446e28c40cfcef0289b87ae9b88" exitCode=0
Aug 04 18:01:44.708563 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:01:44.708163    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-74cb995cd-pjjzg" event={"ID":"4704d0e3-b459-422d-92d9-0e9ec9bbe24f","Type":"ContainerDied","Data":"dd8afcc093487cef90954182a702c1da0525a446e28c40cfcef0289b87ae9b88"}
Aug 04 18:01:44.708563 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:01:44.708175    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-74cb995cd-pjjzg"
Aug 04 18:01:44.708563 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:01:44.708201    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-74cb995cd-pjjzg" event={"ID":"4704d0e3-b459-422d-92d9-0e9ec9bbe24f","Type":"ContainerDied","Data":"28992e5e459c17cb3056ac9b5f022415e1520fb461e65a049a850432fad8a76b"}
Aug 04 18:01:44.708563 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:01:44.708220    2583 scope.go:117] "RemoveContainer" containerID="dd8afcc093487cef90954182a702c1da0525a446e28c40cfcef0289b87ae9b88"
Aug 04 18:01:44.717189 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:01:44.717170    2583 scope.go:117] "RemoveContainer" containerID="dd8afcc093487cef90954182a702c1da0525a446e28c40cfcef0289b87ae9b88"
Aug 04 18:01:44.717439 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:01:44.717421    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd8afcc093487cef90954182a702c1da0525a446e28c40cfcef0289b87ae9b88\": container with ID starting with dd8afcc093487cef90954182a702c1da0525a446e28c40cfcef0289b87ae9b88 not found: ID does not exist" containerID="dd8afcc093487cef90954182a702c1da0525a446e28c40cfcef0289b87ae9b88"
Aug 04 18:01:44.717526 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:01:44.717449    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd8afcc093487cef90954182a702c1da0525a446e28c40cfcef0289b87ae9b88"} err="failed to get container status \"dd8afcc093487cef90954182a702c1da0525a446e28c40cfcef0289b87ae9b88\": rpc error: code = NotFound desc = could not find container \"dd8afcc093487cef90954182a702c1da0525a446e28c40cfcef0289b87ae9b88\": container with ID starting with dd8afcc093487cef90954182a702c1da0525a446e28c40cfcef0289b87ae9b88 not found: ID does not exist"
Aug 04 18:01:44.731898 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:01:44.731877    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-74cb995cd-pjjzg"]
Aug 04 18:01:44.739163 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:01:44.739140    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-74cb995cd-pjjzg"]
Aug 04 18:01:44.943570 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:01:44.943545    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4704d0e3-b459-422d-92d9-0e9ec9bbe24f" path="/var/lib/kubelet/pods/4704d0e3-b459-422d-92d9-0e9ec9bbe24f/volumes"
Aug 04 18:01:52.739615 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:01:52.739581    2583 generic.go:356] "Generic (PLEG): container finished" podID="32978e3c-1e6b-40dc-9dd9-f5bb8f9777ef" containerID="3a98dabd45cd39258c326861f458d29b4998cb96a0816d8c81a6c2c198c0f343" exitCode=0
Aug 04 18:01:52.740061 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:01:52.739657    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-jwhg9" event={"ID":"32978e3c-1e6b-40dc-9dd9-f5bb8f9777ef","Type":"ContainerDied","Data":"3a98dabd45cd39258c326861f458d29b4998cb96a0816d8c81a6c2c198c0f343"}
Aug 04 18:01:52.740061 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:01:52.740042    2583 scope.go:117] "RemoveContainer" containerID="3a98dabd45cd39258c326861f458d29b4998cb96a0816d8c81a6c2c198c0f343"
Aug 04 18:01:53.749619 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:01:53.749582    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-784f97bcf6-jwhg9" event={"ID":"32978e3c-1e6b-40dc-9dd9-f5bb8f9777ef","Type":"ContainerStarted","Data":"57d58fbe07b6187acba52ac095d79fe3668849ea4aa08f80a5914e95a862c5d6"}
Aug 04 18:02:34.096168 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:02:34.096135    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-64d47669bd-nr9rw"]
Aug 04 18:02:34.096647 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:02:34.096411    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4704d0e3-b459-422d-92d9-0e9ec9bbe24f" containerName="registry"
Aug 04 18:02:34.096647 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:02:34.096422    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="4704d0e3-b459-422d-92d9-0e9ec9bbe24f" containerName="registry"
Aug 04 18:02:34.096647 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:02:34.096494    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="4704d0e3-b459-422d-92d9-0e9ec9bbe24f" containerName="registry"
Aug 04 18:02:34.098829 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:02:34.098805    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-64d47669bd-nr9rw"
Aug 04 18:02:34.130151 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:02:34.130126    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-64d47669bd-nr9rw"]
Aug 04 18:02:34.190098 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:02:34.190072    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5c2ebfd2-7cbb-468c-97ee-2867e9cc4b51-service-ca\") pod \"console-64d47669bd-nr9rw\" (UID: \"5c2ebfd2-7cbb-468c-97ee-2867e9cc4b51\") " pod="openshift-console/console-64d47669bd-nr9rw"
Aug 04 18:02:34.190217 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:02:34.190114    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5c2ebfd2-7cbb-468c-97ee-2867e9cc4b51-console-config\") pod \"console-64d47669bd-nr9rw\" (UID: \"5c2ebfd2-7cbb-468c-97ee-2867e9cc4b51\") " pod="openshift-console/console-64d47669bd-nr9rw"
Aug 04 18:02:34.190336 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:02:34.190320    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5c2ebfd2-7cbb-468c-97ee-2867e9cc4b51-console-serving-cert\") pod \"console-64d47669bd-nr9rw\" (UID: \"5c2ebfd2-7cbb-468c-97ee-2867e9cc4b51\") " pod="openshift-console/console-64d47669bd-nr9rw"
Aug 04 18:02:34.190373 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:02:34.190353    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5c2ebfd2-7cbb-468c-97ee-2867e9cc4b51-console-oauth-config\") pod \"console-64d47669bd-nr9rw\" (UID: \"5c2ebfd2-7cbb-468c-97ee-2867e9cc4b51\") " pod="openshift-console/console-64d47669bd-nr9rw"
Aug 04 18:02:34.295916 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:02:34.295873    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5c2ebfd2-7cbb-468c-97ee-2867e9cc4b51-service-ca\") pod \"console-64d47669bd-nr9rw\" (UID: \"5c2ebfd2-7cbb-468c-97ee-2867e9cc4b51\") " pod="openshift-console/console-64d47669bd-nr9rw"
Aug 04 18:02:34.296335 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:02:34.296312    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5c2ebfd2-7cbb-468c-97ee-2867e9cc4b51-console-config\") pod \"console-64d47669bd-nr9rw\" (UID: \"5c2ebfd2-7cbb-468c-97ee-2867e9cc4b51\") " pod="openshift-console/console-64d47669bd-nr9rw"
Aug 04 18:02:34.296409 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:02:34.296382    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5c2ebfd2-7cbb-468c-97ee-2867e9cc4b51-oauth-serving-cert\") pod \"console-64d47669bd-nr9rw\" (UID: \"5c2ebfd2-7cbb-468c-97ee-2867e9cc4b51\") " pod="openshift-console/console-64d47669bd-nr9rw"
Aug 04 18:02:34.296489 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:02:34.296427    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5c2ebfd2-7cbb-468c-97ee-2867e9cc4b51-trusted-ca-bundle\") pod \"console-64d47669bd-nr9rw\" (UID: \"5c2ebfd2-7cbb-468c-97ee-2867e9cc4b51\") " pod="openshift-console/console-64d47669bd-nr9rw"
Aug 04 18:02:34.296546 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:02:34.296520    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5c2ebfd2-7cbb-468c-97ee-2867e9cc4b51-console-serving-cert\") pod \"console-64d47669bd-nr9rw\" (UID: \"5c2ebfd2-7cbb-468c-97ee-2867e9cc4b51\") " pod="openshift-console/console-64d47669bd-nr9rw"
Aug 04 18:02:34.296602 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:02:34.296549    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5fs6\" (UniqueName: \"kubernetes.io/projected/5c2ebfd2-7cbb-468c-97ee-2867e9cc4b51-kube-api-access-m5fs6\") pod \"console-64d47669bd-nr9rw\" (UID: \"5c2ebfd2-7cbb-468c-97ee-2867e9cc4b51\") " pod="openshift-console/console-64d47669bd-nr9rw"
Aug 04 18:02:34.296602 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:02:34.296587    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5c2ebfd2-7cbb-468c-97ee-2867e9cc4b51-console-oauth-config\") pod \"console-64d47669bd-nr9rw\" (UID: \"5c2ebfd2-7cbb-468c-97ee-2867e9cc4b51\") " pod="openshift-console/console-64d47669bd-nr9rw"
Aug 04 18:02:34.296900 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:02:34.296876    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5c2ebfd2-7cbb-468c-97ee-2867e9cc4b51-service-ca\") pod \"console-64d47669bd-nr9rw\" (UID: \"5c2ebfd2-7cbb-468c-97ee-2867e9cc4b51\") " pod="openshift-console/console-64d47669bd-nr9rw"
Aug 04 18:02:34.296958 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:02:34.296934    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5c2ebfd2-7cbb-468c-97ee-2867e9cc4b51-console-config\") pod \"console-64d47669bd-nr9rw\" (UID: \"5c2ebfd2-7cbb-468c-97ee-2867e9cc4b51\") " pod="openshift-console/console-64d47669bd-nr9rw"
Aug 04 18:02:34.298909 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:02:34.298883    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5c2ebfd2-7cbb-468c-97ee-2867e9cc4b51-console-serving-cert\") pod \"console-64d47669bd-nr9rw\" (UID: \"5c2ebfd2-7cbb-468c-97ee-2867e9cc4b51\") " pod="openshift-console/console-64d47669bd-nr9rw"
Aug 04 18:02:34.299029 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:02:34.298914    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5c2ebfd2-7cbb-468c-97ee-2867e9cc4b51-console-oauth-config\") pod \"console-64d47669bd-nr9rw\" (UID: \"5c2ebfd2-7cbb-468c-97ee-2867e9cc4b51\") " pod="openshift-console/console-64d47669bd-nr9rw"
Aug 04 18:02:34.397445 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:02:34.397364    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5fs6\" (UniqueName: \"kubernetes.io/projected/5c2ebfd2-7cbb-468c-97ee-2867e9cc4b51-kube-api-access-m5fs6\") pod \"console-64d47669bd-nr9rw\" (UID: \"5c2ebfd2-7cbb-468c-97ee-2867e9cc4b51\") " pod="openshift-console/console-64d47669bd-nr9rw"
Aug 04 18:02:34.397634 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:02:34.397488    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5c2ebfd2-7cbb-468c-97ee-2867e9cc4b51-oauth-serving-cert\") pod \"console-64d47669bd-nr9rw\" (UID: \"5c2ebfd2-7cbb-468c-97ee-2867e9cc4b51\") " pod="openshift-console/console-64d47669bd-nr9rw"
Aug 04 18:02:34.397634 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:02:34.397521    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5c2ebfd2-7cbb-468c-97ee-2867e9cc4b51-trusted-ca-bundle\") pod \"console-64d47669bd-nr9rw\" (UID: \"5c2ebfd2-7cbb-468c-97ee-2867e9cc4b51\") " pod="openshift-console/console-64d47669bd-nr9rw"
Aug 04 18:02:34.398093 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:02:34.398070    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5c2ebfd2-7cbb-468c-97ee-2867e9cc4b51-oauth-serving-cert\") pod \"console-64d47669bd-nr9rw\" (UID: \"5c2ebfd2-7cbb-468c-97ee-2867e9cc4b51\") " pod="openshift-console/console-64d47669bd-nr9rw"
Aug 04 18:02:34.398412 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:02:34.398393    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5c2ebfd2-7cbb-468c-97ee-2867e9cc4b51-trusted-ca-bundle\") pod \"console-64d47669bd-nr9rw\" (UID: \"5c2ebfd2-7cbb-468c-97ee-2867e9cc4b51\") " pod="openshift-console/console-64d47669bd-nr9rw"
Aug 04 18:02:34.407279 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:02:34.407256    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5fs6\" (UniqueName: \"kubernetes.io/projected/5c2ebfd2-7cbb-468c-97ee-2867e9cc4b51-kube-api-access-m5fs6\") pod \"console-64d47669bd-nr9rw\" (UID: \"5c2ebfd2-7cbb-468c-97ee-2867e9cc4b51\") " pod="openshift-console/console-64d47669bd-nr9rw"
Aug 04 18:02:34.408099 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:02:34.408070    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-64d47669bd-nr9rw"
Aug 04 18:02:34.553319 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:02:34.553289    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-64d47669bd-nr9rw"]
Aug 04 18:02:34.556085 ip-10-0-135-124 kubenswrapper[2583]: W0804 18:02:34.556061    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c2ebfd2_7cbb_468c_97ee_2867e9cc4b51.slice/crio-2b9b311de79b1d68ebf24d947a1ac60ca72713868f8751c026196b3f5463c5b1 WatchSource:0}: Error finding container 2b9b311de79b1d68ebf24d947a1ac60ca72713868f8751c026196b3f5463c5b1: Status 404 returned error can't find the container with id 2b9b311de79b1d68ebf24d947a1ac60ca72713868f8751c026196b3f5463c5b1
Aug 04 18:02:34.905952 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:02:34.905919    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-64d47669bd-nr9rw" event={"ID":"5c2ebfd2-7cbb-468c-97ee-2867e9cc4b51","Type":"ContainerStarted","Data":"1c60923780a402052b0efd4fe0c8a674d906a8041766e29a302f84b2e9b980ce"}
Aug 04 18:02:34.906122 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:02:34.905966    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-64d47669bd-nr9rw" event={"ID":"5c2ebfd2-7cbb-468c-97ee-2867e9cc4b51","Type":"ContainerStarted","Data":"2b9b311de79b1d68ebf24d947a1ac60ca72713868f8751c026196b3f5463c5b1"}
Aug 04 18:02:34.930089 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:02:34.930036    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-64d47669bd-nr9rw" podStartSLOduration=0.930021142 podStartE2EDuration="930.021142ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:02:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:02:34.928841511 +0000 UTC m=+168.510060089" watchObservedRunningTime="2026-08-04 18:02:34.930021142 +0000 UTC m=+168.511239719"
Aug 04 18:02:44.408582 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:02:44.408542    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-64d47669bd-nr9rw"
Aug 04 18:02:44.409002 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:02:44.408596    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-64d47669bd-nr9rw"
Aug 04 18:02:44.413555 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:02:44.413533    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-64d47669bd-nr9rw"
Aug 04 18:02:44.949267 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:02:44.949235    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-64d47669bd-nr9rw"
Aug 04 18:02:45.008790 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:02:45.008762    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7ffcf99db4-k6vvm"]
Aug 04 18:03:10.028667 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:03:10.028612    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7ffcf99db4-k6vvm" podUID="144865c4-61dd-4fa5-b40d-f08e404fdc9d" containerName="console" containerID="cri-o://d405ac3a7194f501d20d5c1aa762f04b164949287bbe36b0e31fd0e1a0f95bb4" gracePeriod=15
Aug 04 18:03:10.260775 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:03:10.260750    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7ffcf99db4-k6vvm"
Aug 04 18:03:10.377761 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:03:10.377681    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/144865c4-61dd-4fa5-b40d-f08e404fdc9d-trusted-ca-bundle\") pod \"144865c4-61dd-4fa5-b40d-f08e404fdc9d\" (UID: \"144865c4-61dd-4fa5-b40d-f08e404fdc9d\") "
Aug 04 18:03:10.377761 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:03:10.377749    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/144865c4-61dd-4fa5-b40d-f08e404fdc9d-console-oauth-config\") pod \"144865c4-61dd-4fa5-b40d-f08e404fdc9d\" (UID: \"144865c4-61dd-4fa5-b40d-f08e404fdc9d\") "
Aug 04 18:03:10.377985 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:03:10.377784    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/144865c4-61dd-4fa5-b40d-f08e404fdc9d-service-ca\") pod \"144865c4-61dd-4fa5-b40d-f08e404fdc9d\" (UID: \"144865c4-61dd-4fa5-b40d-f08e404fdc9d\") "
Aug 04 18:03:10.377985 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:03:10.377811    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v7hpc\" (UniqueName: \"kubernetes.io/projected/144865c4-61dd-4fa5-b40d-f08e404fdc9d-kube-api-access-v7hpc\") pod \"144865c4-61dd-4fa5-b40d-f08e404fdc9d\" (UID: \"144865c4-61dd-4fa5-b40d-f08e404fdc9d\") "
Aug 04 18:03:10.377985 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:03:10.377844    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/144865c4-61dd-4fa5-b40d-f08e404fdc9d-console-config\") pod \"144865c4-61dd-4fa5-b40d-f08e404fdc9d\" (UID: \"144865c4-61dd-4fa5-b40d-f08e404fdc9d\") "
Aug 04 18:03:10.377985 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:03:10.377869    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/144865c4-61dd-4fa5-b40d-f08e404fdc9d-oauth-serving-cert\") pod \"144865c4-61dd-4fa5-b40d-f08e404fdc9d\" (UID: \"144865c4-61dd-4fa5-b40d-f08e404fdc9d\") "
Aug 04 18:03:10.377985 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:03:10.377932    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/144865c4-61dd-4fa5-b40d-f08e404fdc9d-console-serving-cert\") pod \"144865c4-61dd-4fa5-b40d-f08e404fdc9d\" (UID: \"144865c4-61dd-4fa5-b40d-f08e404fdc9d\") "
Aug 04 18:03:10.378298 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:03:10.378266    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/144865c4-61dd-4fa5-b40d-f08e404fdc9d-service-ca" (OuterVolumeSpecName: "service-ca") pod "144865c4-61dd-4fa5-b40d-f08e404fdc9d" (UID: "144865c4-61dd-4fa5-b40d-f08e404fdc9d"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:03:10.378377 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:03:10.378294    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/144865c4-61dd-4fa5-b40d-f08e404fdc9d-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "144865c4-61dd-4fa5-b40d-f08e404fdc9d" (UID: "144865c4-61dd-4fa5-b40d-f08e404fdc9d"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:03:10.378377 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:03:10.378316    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/144865c4-61dd-4fa5-b40d-f08e404fdc9d-console-config" (OuterVolumeSpecName: "console-config") pod "144865c4-61dd-4fa5-b40d-f08e404fdc9d" (UID: "144865c4-61dd-4fa5-b40d-f08e404fdc9d"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:03:10.378377 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:03:10.378345    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/144865c4-61dd-4fa5-b40d-f08e404fdc9d-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "144865c4-61dd-4fa5-b40d-f08e404fdc9d" (UID: "144865c4-61dd-4fa5-b40d-f08e404fdc9d"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:03:10.380059 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:03:10.380034    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/144865c4-61dd-4fa5-b40d-f08e404fdc9d-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "144865c4-61dd-4fa5-b40d-f08e404fdc9d" (UID: "144865c4-61dd-4fa5-b40d-f08e404fdc9d"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:03:10.380145 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:03:10.380068    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/144865c4-61dd-4fa5-b40d-f08e404fdc9d-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "144865c4-61dd-4fa5-b40d-f08e404fdc9d" (UID: "144865c4-61dd-4fa5-b40d-f08e404fdc9d"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:03:10.380183 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:03:10.380148    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/144865c4-61dd-4fa5-b40d-f08e404fdc9d-kube-api-access-v7hpc" (OuterVolumeSpecName: "kube-api-access-v7hpc") pod "144865c4-61dd-4fa5-b40d-f08e404fdc9d" (UID: "144865c4-61dd-4fa5-b40d-f08e404fdc9d"). InnerVolumeSpecName "kube-api-access-v7hpc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:03:10.478650 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:03:10.478615    2583 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/144865c4-61dd-4fa5-b40d-f08e404fdc9d-console-oauth-config\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:03:10.478650 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:03:10.478646    2583 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/144865c4-61dd-4fa5-b40d-f08e404fdc9d-service-ca\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:03:10.478650 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:03:10.478656    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v7hpc\" (UniqueName: \"kubernetes.io/projected/144865c4-61dd-4fa5-b40d-f08e404fdc9d-kube-api-access-v7hpc\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:03:10.478813 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:03:10.478665    2583 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/144865c4-61dd-4fa5-b40d-f08e404fdc9d-console-config\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:03:10.478813 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:03:10.478673    2583 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/144865c4-61dd-4fa5-b40d-f08e404fdc9d-oauth-serving-cert\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:03:10.478813 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:03:10.478681    2583 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/144865c4-61dd-4fa5-b40d-f08e404fdc9d-console-serving-cert\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:03:10.478813 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:03:10.478688    2583 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/144865c4-61dd-4fa5-b40d-f08e404fdc9d-trusted-ca-bundle\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:03:11.040816 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:03:11.040789    2583 generic.go:356] "Generic (PLEG): container finished" podID="144865c4-61dd-4fa5-b40d-f08e404fdc9d" containerID="d405ac3a7194f501d20d5c1aa762f04b164949287bbe36b0e31fd0e1a0f95bb4" exitCode=0
Aug 04 18:03:11.041234 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:03:11.040832    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7ffcf99db4-k6vvm"
Aug 04 18:03:11.041234 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:03:11.040877    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7ffcf99db4-k6vvm" event={"ID":"144865c4-61dd-4fa5-b40d-f08e404fdc9d","Type":"ContainerDied","Data":"d405ac3a7194f501d20d5c1aa762f04b164949287bbe36b0e31fd0e1a0f95bb4"}
Aug 04 18:03:11.041234 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:03:11.040932    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7ffcf99db4-k6vvm" event={"ID":"144865c4-61dd-4fa5-b40d-f08e404fdc9d","Type":"ContainerDied","Data":"dcc43f27a54293014813e71c70029d7e69897668df65ffe1e83a0b23f582b630"}
Aug 04 18:03:11.041234 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:03:11.040963    2583 scope.go:117] "RemoveContainer" containerID="d405ac3a7194f501d20d5c1aa762f04b164949287bbe36b0e31fd0e1a0f95bb4"
Aug 04 18:03:11.048886 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:03:11.048870    2583 scope.go:117] "RemoveContainer" containerID="d405ac3a7194f501d20d5c1aa762f04b164949287bbe36b0e31fd0e1a0f95bb4"
Aug 04 18:03:11.049167 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:03:11.049149    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d405ac3a7194f501d20d5c1aa762f04b164949287bbe36b0e31fd0e1a0f95bb4\": container with ID starting with d405ac3a7194f501d20d5c1aa762f04b164949287bbe36b0e31fd0e1a0f95bb4 not found: ID does not exist" containerID="d405ac3a7194f501d20d5c1aa762f04b164949287bbe36b0e31fd0e1a0f95bb4"
Aug 04 18:03:11.049212 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:03:11.049174    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d405ac3a7194f501d20d5c1aa762f04b164949287bbe36b0e31fd0e1a0f95bb4"} err="failed to get container status \"d405ac3a7194f501d20d5c1aa762f04b164949287bbe36b0e31fd0e1a0f95bb4\": rpc error: code = NotFound desc = could not find container \"d405ac3a7194f501d20d5c1aa762f04b164949287bbe36b0e31fd0e1a0f95bb4\": container with ID starting with d405ac3a7194f501d20d5c1aa762f04b164949287bbe36b0e31fd0e1a0f95bb4 not found: ID does not exist"
Aug 04 18:03:11.059321 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:03:11.059301    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7ffcf99db4-k6vvm"]
Aug 04 18:03:11.063603 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:03:11.063584    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7ffcf99db4-k6vvm"]
Aug 04 18:03:12.940580 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:03:12.940549    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="144865c4-61dd-4fa5-b40d-f08e404fdc9d" path="/var/lib/kubelet/pods/144865c4-61dd-4fa5-b40d-f08e404fdc9d/volumes"
Aug 04 18:04:13.624147 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:04:13.624068    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b55snpn"]
Aug 04 18:04:13.624566 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:04:13.624348    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="144865c4-61dd-4fa5-b40d-f08e404fdc9d" containerName="console"
Aug 04 18:04:13.624566 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:04:13.624360    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="144865c4-61dd-4fa5-b40d-f08e404fdc9d" containerName="console"
Aug 04 18:04:13.624566 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:04:13.624417    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="144865c4-61dd-4fa5-b40d-f08e404fdc9d" containerName="console"
Aug 04 18:04:13.626230 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:04:13.626212    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b55snpn"
Aug 04 18:04:13.629054 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:04:13.629022    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-wf5qp\""
Aug 04 18:04:13.629209 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:04:13.629022    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Aug 04 18:04:13.630085 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:04:13.630067    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Aug 04 18:04:13.636092 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:04:13.636070    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b55snpn"]
Aug 04 18:04:13.729111 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:04:13.729071    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6f234f7f-1d17-40f5-8325-4b9a8211b23f-bundle\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b55snpn\" (UID: \"6f234f7f-1d17-40f5-8325-4b9a8211b23f\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b55snpn"
Aug 04 18:04:13.729309 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:04:13.729137    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6f234f7f-1d17-40f5-8325-4b9a8211b23f-util\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b55snpn\" (UID: \"6f234f7f-1d17-40f5-8325-4b9a8211b23f\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b55snpn"
Aug 04 18:04:13.729309 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:04:13.729205    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rnhd\" (UniqueName: \"kubernetes.io/projected/6f234f7f-1d17-40f5-8325-4b9a8211b23f-kube-api-access-4rnhd\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b55snpn\" (UID: \"6f234f7f-1d17-40f5-8325-4b9a8211b23f\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b55snpn"
Aug 04 18:04:13.829652 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:04:13.829605    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6f234f7f-1d17-40f5-8325-4b9a8211b23f-bundle\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b55snpn\" (UID: \"6f234f7f-1d17-40f5-8325-4b9a8211b23f\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b55snpn"
Aug 04 18:04:13.829805 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:04:13.829684    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6f234f7f-1d17-40f5-8325-4b9a8211b23f-util\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b55snpn\" (UID: \"6f234f7f-1d17-40f5-8325-4b9a8211b23f\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b55snpn"
Aug 04 18:04:13.829805 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:04:13.829713    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rnhd\" (UniqueName: \"kubernetes.io/projected/6f234f7f-1d17-40f5-8325-4b9a8211b23f-kube-api-access-4rnhd\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b55snpn\" (UID: \"6f234f7f-1d17-40f5-8325-4b9a8211b23f\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b55snpn"
Aug 04 18:04:13.830078 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:04:13.830057    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6f234f7f-1d17-40f5-8325-4b9a8211b23f-bundle\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b55snpn\" (UID: \"6f234f7f-1d17-40f5-8325-4b9a8211b23f\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b55snpn"
Aug 04 18:04:13.830167 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:04:13.830057    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6f234f7f-1d17-40f5-8325-4b9a8211b23f-util\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b55snpn\" (UID: \"6f234f7f-1d17-40f5-8325-4b9a8211b23f\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b55snpn"
Aug 04 18:04:13.838695 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:04:13.838674    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rnhd\" (UniqueName: \"kubernetes.io/projected/6f234f7f-1d17-40f5-8325-4b9a8211b23f-kube-api-access-4rnhd\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b55snpn\" (UID: \"6f234f7f-1d17-40f5-8325-4b9a8211b23f\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b55snpn"
Aug 04 18:04:13.935697 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:04:13.935669    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b55snpn"
Aug 04 18:04:14.062929 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:04:14.062895    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b55snpn"]
Aug 04 18:04:14.065590 ip-10-0-135-124 kubenswrapper[2583]: W0804 18:04:14.065564    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f234f7f_1d17_40f5_8325_4b9a8211b23f.slice/crio-c249a476ab79243f60a7bcc156e5a38fe0eafaa200b190b2550506f794460817 WatchSource:0}: Error finding container c249a476ab79243f60a7bcc156e5a38fe0eafaa200b190b2550506f794460817: Status 404 returned error can't find the container with id c249a476ab79243f60a7bcc156e5a38fe0eafaa200b190b2550506f794460817
Aug 04 18:04:14.268494 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:04:14.268389    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b55snpn" event={"ID":"6f234f7f-1d17-40f5-8325-4b9a8211b23f","Type":"ContainerStarted","Data":"c249a476ab79243f60a7bcc156e5a38fe0eafaa200b190b2550506f794460817"}
Aug 04 18:04:22.302633 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:04:22.302595    2583 generic.go:356] "Generic (PLEG): container finished" podID="6f234f7f-1d17-40f5-8325-4b9a8211b23f" containerID="bf5cf941b1405cfc8cff2bb31c91cb1575880acbab810a310dd0ea5a6067d012" exitCode=0
Aug 04 18:04:22.303059 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:04:22.302685    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b55snpn" event={"ID":"6f234f7f-1d17-40f5-8325-4b9a8211b23f","Type":"ContainerDied","Data":"bf5cf941b1405cfc8cff2bb31c91cb1575880acbab810a310dd0ea5a6067d012"}
Aug 04 18:04:25.315717 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:04:25.315686    2583 generic.go:356] "Generic (PLEG): container finished" podID="6f234f7f-1d17-40f5-8325-4b9a8211b23f" containerID="4b60b125fbcec32e519f722ea33efda2d783506bc54cab765e19be861a2a9ee6" exitCode=0
Aug 04 18:04:25.316146 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:04:25.315770    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b55snpn" event={"ID":"6f234f7f-1d17-40f5-8325-4b9a8211b23f","Type":"ContainerDied","Data":"4b60b125fbcec32e519f722ea33efda2d783506bc54cab765e19be861a2a9ee6"}
Aug 04 18:04:32.346097 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:04:32.346012    2583 generic.go:356] "Generic (PLEG): container finished" podID="6f234f7f-1d17-40f5-8325-4b9a8211b23f" containerID="7d56e0e6d47a678fe4e75dd0edd04274f13bab3789549a05a44a9201cd155723" exitCode=0
Aug 04 18:04:32.346097 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:04:32.346069    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b55snpn" event={"ID":"6f234f7f-1d17-40f5-8325-4b9a8211b23f","Type":"ContainerDied","Data":"7d56e0e6d47a678fe4e75dd0edd04274f13bab3789549a05a44a9201cd155723"}
Aug 04 18:04:33.474963 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:04:33.474938    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b55snpn"
Aug 04 18:04:33.596368 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:04:33.596338    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6f234f7f-1d17-40f5-8325-4b9a8211b23f-bundle\") pod \"6f234f7f-1d17-40f5-8325-4b9a8211b23f\" (UID: \"6f234f7f-1d17-40f5-8325-4b9a8211b23f\") "
Aug 04 18:04:33.596541 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:04:33.596431    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4rnhd\" (UniqueName: \"kubernetes.io/projected/6f234f7f-1d17-40f5-8325-4b9a8211b23f-kube-api-access-4rnhd\") pod \"6f234f7f-1d17-40f5-8325-4b9a8211b23f\" (UID: \"6f234f7f-1d17-40f5-8325-4b9a8211b23f\") "
Aug 04 18:04:33.596541 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:04:33.596506    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6f234f7f-1d17-40f5-8325-4b9a8211b23f-util\") pod \"6f234f7f-1d17-40f5-8325-4b9a8211b23f\" (UID: \"6f234f7f-1d17-40f5-8325-4b9a8211b23f\") "
Aug 04 18:04:33.597026 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:04:33.597001    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f234f7f-1d17-40f5-8325-4b9a8211b23f-bundle" (OuterVolumeSpecName: "bundle") pod "6f234f7f-1d17-40f5-8325-4b9a8211b23f" (UID: "6f234f7f-1d17-40f5-8325-4b9a8211b23f"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:04:33.598522 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:04:33.598499    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f234f7f-1d17-40f5-8325-4b9a8211b23f-kube-api-access-4rnhd" (OuterVolumeSpecName: "kube-api-access-4rnhd") pod "6f234f7f-1d17-40f5-8325-4b9a8211b23f" (UID: "6f234f7f-1d17-40f5-8325-4b9a8211b23f"). InnerVolumeSpecName "kube-api-access-4rnhd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:04:33.601029 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:04:33.600976    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f234f7f-1d17-40f5-8325-4b9a8211b23f-util" (OuterVolumeSpecName: "util") pod "6f234f7f-1d17-40f5-8325-4b9a8211b23f" (UID: "6f234f7f-1d17-40f5-8325-4b9a8211b23f"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:04:33.697370 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:04:33.697347    2583 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6f234f7f-1d17-40f5-8325-4b9a8211b23f-bundle\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:04:33.697370 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:04:33.697367    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4rnhd\" (UniqueName: \"kubernetes.io/projected/6f234f7f-1d17-40f5-8325-4b9a8211b23f-kube-api-access-4rnhd\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:04:33.697498 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:04:33.697377    2583 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6f234f7f-1d17-40f5-8325-4b9a8211b23f-util\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:04:34.354694 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:04:34.354664    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b55snpn"
Aug 04 18:04:34.354853 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:04:34.354697    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b55snpn" event={"ID":"6f234f7f-1d17-40f5-8325-4b9a8211b23f","Type":"ContainerDied","Data":"c249a476ab79243f60a7bcc156e5a38fe0eafaa200b190b2550506f794460817"}
Aug 04 18:04:34.354853 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:04:34.354752    2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c249a476ab79243f60a7bcc156e5a38fe0eafaa200b190b2550506f794460817"
Aug 04 18:04:41.377987 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:04:41.377957    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-4jp2h"]
Aug 04 18:04:41.378377 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:04:41.378264    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f234f7f-1d17-40f5-8325-4b9a8211b23f" containerName="pull"
Aug 04 18:04:41.378377 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:04:41.378276    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f234f7f-1d17-40f5-8325-4b9a8211b23f" containerName="pull"
Aug 04 18:04:41.378377 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:04:41.378284    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f234f7f-1d17-40f5-8325-4b9a8211b23f" containerName="extract"
Aug 04 18:04:41.378377 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:04:41.378290    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f234f7f-1d17-40f5-8325-4b9a8211b23f" containerName="extract"
Aug 04 18:04:41.378377 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:04:41.378306    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f234f7f-1d17-40f5-8325-4b9a8211b23f" containerName="util"
Aug 04 18:04:41.378377 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:04:41.378313    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f234f7f-1d17-40f5-8325-4b9a8211b23f" containerName="util"
Aug 04 18:04:41.378377 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:04:41.378370    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f234f7f-1d17-40f5-8325-4b9a8211b23f" containerName="extract"
Aug 04 18:04:41.380149 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:04:41.380134    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-4jp2h"
Aug 04 18:04:41.383488 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:04:41.383441    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Aug 04 18:04:41.383488 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:04:41.383460    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"custom-metrics-autoscaler-operator-dockercfg-97jwq\""
Aug 04 18:04:41.383676 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:04:41.383460    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Aug 04 18:04:41.383676 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:04:41.383545    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Aug 04 18:04:41.396510 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:04:41.396485    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-4jp2h"]
Aug 04 18:04:41.452928 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:04:41.452904    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/0580f10d-c147-4c3b-a5a7-0d2401b03602-certificates\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-4jp2h\" (UID: \"0580f10d-c147-4c3b-a5a7-0d2401b03602\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-4jp2h"
Aug 04 18:04:41.452928 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:04:41.452935    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdwcx\" (UniqueName: \"kubernetes.io/projected/0580f10d-c147-4c3b-a5a7-0d2401b03602-kube-api-access-hdwcx\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-4jp2h\" (UID: \"0580f10d-c147-4c3b-a5a7-0d2401b03602\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-4jp2h"
Aug 04 18:04:41.554145 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:04:41.554118    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/0580f10d-c147-4c3b-a5a7-0d2401b03602-certificates\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-4jp2h\" (UID: \"0580f10d-c147-4c3b-a5a7-0d2401b03602\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-4jp2h"
Aug 04 18:04:41.554262 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:04:41.554152    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdwcx\" (UniqueName: \"kubernetes.io/projected/0580f10d-c147-4c3b-a5a7-0d2401b03602-kube-api-access-hdwcx\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-4jp2h\" (UID: \"0580f10d-c147-4c3b-a5a7-0d2401b03602\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-4jp2h"
Aug 04 18:04:41.556462 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:04:41.556435    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/0580f10d-c147-4c3b-a5a7-0d2401b03602-certificates\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-4jp2h\" (UID: \"0580f10d-c147-4c3b-a5a7-0d2401b03602\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-4jp2h"
Aug 04 18:04:41.563716 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:04:41.563695    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdwcx\" (UniqueName: \"kubernetes.io/projected/0580f10d-c147-4c3b-a5a7-0d2401b03602-kube-api-access-hdwcx\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-4jp2h\" (UID: \"0580f10d-c147-4c3b-a5a7-0d2401b03602\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-4jp2h"
Aug 04 18:04:41.689958 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:04:41.689924    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-4jp2h"
Aug 04 18:04:41.820632 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:04:41.820568    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-4jp2h"]
Aug 04 18:04:41.822973 ip-10-0-135-124 kubenswrapper[2583]: W0804 18:04:41.822945    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0580f10d_c147_4c3b_a5a7_0d2401b03602.slice/crio-85aa6c661ea624dce28439bcd21b54caceb584eae1913a9dec544356a9c46cc3 WatchSource:0}: Error finding container 85aa6c661ea624dce28439bcd21b54caceb584eae1913a9dec544356a9c46cc3: Status 404 returned error can't find the container with id 85aa6c661ea624dce28439bcd21b54caceb584eae1913a9dec544356a9c46cc3
Aug 04 18:04:42.385354 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:04:42.385319    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-4jp2h" event={"ID":"0580f10d-c147-4c3b-a5a7-0d2401b03602","Type":"ContainerStarted","Data":"85aa6c661ea624dce28439bcd21b54caceb584eae1913a9dec544356a9c46cc3"}
Aug 04 18:04:46.328582 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:04:46.328544    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-operator-77f648d694-fvmgq"]
Aug 04 18:04:46.330694 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:04:46.330674    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-77f648d694-fvmgq"
Aug 04 18:04:46.333615 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:04:46.333591    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-q6h52\""
Aug 04 18:04:46.334653 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:04:46.334632    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-certs\""
Aug 04 18:04:46.334745 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:04:46.334668    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Aug 04 18:04:46.343077 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:04:46.343046    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-77f648d694-fvmgq"]
Aug 04 18:04:46.403805 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:04:46.403778    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-4jp2h" event={"ID":"0580f10d-c147-4c3b-a5a7-0d2401b03602","Type":"ContainerStarted","Data":"2386ab2e8ad55e7996596372309de2f67fb50612128e2ecf7f4936b1b1dd401f"}
Aug 04 18:04:46.403932 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:04:46.403904    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-4jp2h"
Aug 04 18:04:46.423894 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:04:46.423851    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-4jp2h" podStartSLOduration=1.6839979029999999 podStartE2EDuration="5.423836913s" totalImagesPullingTime="3.73983901s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:04:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:04:46.42345711 +0000 UTC m=+300.004675687" watchObservedRunningTime="2026-08-04 18:04:46.423836913 +0000 UTC m=+300.005055491"
Aug 04 18:04:46.497379 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:04:46.497350    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bwd9\" (UniqueName: \"kubernetes.io/projected/e35cd866-af62-4a66-bb6d-7ff46fa93439-kube-api-access-8bwd9\") pod \"keda-operator-77f648d694-fvmgq\" (UID: \"e35cd866-af62-4a66-bb6d-7ff46fa93439\") " pod="openshift-keda/keda-operator-77f648d694-fvmgq"
Aug 04 18:04:46.497534 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:04:46.497516    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/e35cd866-af62-4a66-bb6d-7ff46fa93439-certificates\") pod \"keda-operator-77f648d694-fvmgq\" (UID: \"e35cd866-af62-4a66-bb6d-7ff46fa93439\") " pod="openshift-keda/keda-operator-77f648d694-fvmgq"
Aug 04 18:04:46.497624 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:04:46.497555    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/e35cd866-af62-4a66-bb6d-7ff46fa93439-cabundle0\") pod \"keda-operator-77f648d694-fvmgq\" (UID: \"e35cd866-af62-4a66-bb6d-7ff46fa93439\") " pod="openshift-keda/keda-operator-77f648d694-fvmgq"
Aug 04 18:04:46.534763 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:04:46.534736    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-metrics-apiserver-55557df8c-pjbv9"]
Aug 04 18:04:46.537373 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:04:46.537356    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-55557df8c-pjbv9"
Aug 04 18:04:46.539811 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:04:46.539795    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-metrics-apiserver-certs\""
Aug 04 18:04:46.547458 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:04:46.547436    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-55557df8c-pjbv9"]
Aug 04 18:04:46.598569 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:04:46.598512    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/e35cd866-af62-4a66-bb6d-7ff46fa93439-certificates\") pod \"keda-operator-77f648d694-fvmgq\" (UID: \"e35cd866-af62-4a66-bb6d-7ff46fa93439\") " pod="openshift-keda/keda-operator-77f648d694-fvmgq"
Aug 04 18:04:46.598569 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:04:46.598542    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/e35cd866-af62-4a66-bb6d-7ff46fa93439-cabundle0\") pod \"keda-operator-77f648d694-fvmgq\" (UID: \"e35cd866-af62-4a66-bb6d-7ff46fa93439\") " pod="openshift-keda/keda-operator-77f648d694-fvmgq"
Aug 04 18:04:46.598705 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:04:46.598595    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bwd9\" (UniqueName: \"kubernetes.io/projected/e35cd866-af62-4a66-bb6d-7ff46fa93439-kube-api-access-8bwd9\") pod \"keda-operator-77f648d694-fvmgq\" (UID: \"e35cd866-af62-4a66-bb6d-7ff46fa93439\") " pod="openshift-keda/keda-operator-77f648d694-fvmgq"
Aug 04 18:04:46.599281 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:04:46.599263    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/e35cd866-af62-4a66-bb6d-7ff46fa93439-cabundle0\") pod \"keda-operator-77f648d694-fvmgq\" (UID: \"e35cd866-af62-4a66-bb6d-7ff46fa93439\") " pod="openshift-keda/keda-operator-77f648d694-fvmgq"
Aug 04 18:04:46.600857 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:04:46.600840    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/e35cd866-af62-4a66-bb6d-7ff46fa93439-certificates\") pod \"keda-operator-77f648d694-fvmgq\" (UID: \"e35cd866-af62-4a66-bb6d-7ff46fa93439\") " pod="openshift-keda/keda-operator-77f648d694-fvmgq"
Aug 04 18:04:46.606833 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:04:46.606815    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bwd9\" (UniqueName: \"kubernetes.io/projected/e35cd866-af62-4a66-bb6d-7ff46fa93439-kube-api-access-8bwd9\") pod \"keda-operator-77f648d694-fvmgq\" (UID: \"e35cd866-af62-4a66-bb6d-7ff46fa93439\") " pod="openshift-keda/keda-operator-77f648d694-fvmgq"
Aug 04 18:04:46.640692 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:04:46.640669    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-77f648d694-fvmgq"
Aug 04 18:04:46.699951 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:04:46.699918    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mxzg\" (UniqueName: \"kubernetes.io/projected/0121a4cc-ab1c-4949-8c5b-b205ec974874-kube-api-access-5mxzg\") pod \"keda-metrics-apiserver-55557df8c-pjbv9\" (UID: \"0121a4cc-ab1c-4949-8c5b-b205ec974874\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-pjbv9"
Aug 04 18:04:46.700092 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:04:46.700002    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/0121a4cc-ab1c-4949-8c5b-b205ec974874-temp-vol\") pod \"keda-metrics-apiserver-55557df8c-pjbv9\" (UID: \"0121a4cc-ab1c-4949-8c5b-b205ec974874\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-pjbv9"
Aug 04 18:04:46.700092 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:04:46.700058    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/0121a4cc-ab1c-4949-8c5b-b205ec974874-certificates\") pod \"keda-metrics-apiserver-55557df8c-pjbv9\" (UID: \"0121a4cc-ab1c-4949-8c5b-b205ec974874\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-pjbv9"
Aug 04 18:04:46.775784 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:04:46.775704    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-77f648d694-fvmgq"]
Aug 04 18:04:46.777994 ip-10-0-135-124 kubenswrapper[2583]: W0804 18:04:46.777969    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode35cd866_af62_4a66_bb6d_7ff46fa93439.slice/crio-400a5e846bc454eb06d7b530043e0d12b62dec4b5f08bf1d14d9e563665688f3 WatchSource:0}: Error finding container 400a5e846bc454eb06d7b530043e0d12b62dec4b5f08bf1d14d9e563665688f3: Status 404 returned error can't find the container with id 400a5e846bc454eb06d7b530043e0d12b62dec4b5f08bf1d14d9e563665688f3
Aug 04 18:04:46.790009 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:04:46.789983    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-admission-56f9798bb4-pttmg"]
Aug 04 18:04:46.792575 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:04:46.792559    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-56f9798bb4-pttmg"
Aug 04 18:04:46.795637 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:04:46.795621    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-admission-webhooks-certs\""
Aug 04 18:04:46.800551 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:04:46.800525    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mxzg\" (UniqueName: \"kubernetes.io/projected/0121a4cc-ab1c-4949-8c5b-b205ec974874-kube-api-access-5mxzg\") pod \"keda-metrics-apiserver-55557df8c-pjbv9\" (UID: \"0121a4cc-ab1c-4949-8c5b-b205ec974874\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-pjbv9"
Aug 04 18:04:46.800861 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:04:46.800622    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/0121a4cc-ab1c-4949-8c5b-b205ec974874-temp-vol\") pod \"keda-metrics-apiserver-55557df8c-pjbv9\" (UID: \"0121a4cc-ab1c-4949-8c5b-b205ec974874\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-pjbv9"
Aug 04 18:04:46.800861 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:04:46.800689    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/0121a4cc-ab1c-4949-8c5b-b205ec974874-certificates\") pod \"keda-metrics-apiserver-55557df8c-pjbv9\" (UID: \"0121a4cc-ab1c-4949-8c5b-b205ec974874\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-pjbv9"
Aug 04 18:04:46.801253 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:04:46.801227    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/0121a4cc-ab1c-4949-8c5b-b205ec974874-temp-vol\") pod \"keda-metrics-apiserver-55557df8c-pjbv9\" (UID: \"0121a4cc-ab1c-4949-8c5b-b205ec974874\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-pjbv9"
Aug 04 18:04:46.803153 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:04:46.803131    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-56f9798bb4-pttmg"]
Aug 04 18:04:46.803479 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:04:46.803445    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/0121a4cc-ab1c-4949-8c5b-b205ec974874-certificates\") pod \"keda-metrics-apiserver-55557df8c-pjbv9\" (UID: \"0121a4cc-ab1c-4949-8c5b-b205ec974874\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-pjbv9"
Aug 04 18:04:46.812210 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:04:46.812190    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mxzg\" (UniqueName: \"kubernetes.io/projected/0121a4cc-ab1c-4949-8c5b-b205ec974874-kube-api-access-5mxzg\") pod \"keda-metrics-apiserver-55557df8c-pjbv9\" (UID: \"0121a4cc-ab1c-4949-8c5b-b205ec974874\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-pjbv9"
Aug 04 18:04:46.845279 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:04:46.845255    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-wskfb_6157c2d3-2fd0-4544-88ad-434ac75f8c04/console-operator/1.log"
Aug 04 18:04:46.846229 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:04:46.846156    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-wskfb_6157c2d3-2fd0-4544-88ad-434ac75f8c04/console-operator/1.log"
Aug 04 18:04:46.850363 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:04:46.850178    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-55557df8c-pjbv9"
Aug 04 18:04:46.859613 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:04:46.859593    2583 kubelet.go:1648] "Image garbage collection succeeded"
Aug 04 18:04:46.908390 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:04:46.902034    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hg27p\" (UniqueName: \"kubernetes.io/projected/2a17f495-bbf8-4ecf-bbc5-cf9b4f32bcd7-kube-api-access-hg27p\") pod \"keda-admission-56f9798bb4-pttmg\" (UID: \"2a17f495-bbf8-4ecf-bbc5-cf9b4f32bcd7\") " pod="openshift-keda/keda-admission-56f9798bb4-pttmg"
Aug 04 18:04:46.908390 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:04:46.902108    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/2a17f495-bbf8-4ecf-bbc5-cf9b4f32bcd7-certificates\") pod \"keda-admission-56f9798bb4-pttmg\" (UID: \"2a17f495-bbf8-4ecf-bbc5-cf9b4f32bcd7\") " pod="openshift-keda/keda-admission-56f9798bb4-pttmg"
Aug 04 18:04:46.980488 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:04:46.980452    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-55557df8c-pjbv9"]
Aug 04 18:04:46.982668 ip-10-0-135-124 kubenswrapper[2583]: W0804 18:04:46.982646    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0121a4cc_ab1c_4949_8c5b_b205ec974874.slice/crio-be10be330665131af8f733d199550342f3d278a7f6a48a574f2b911ac5ff742c WatchSource:0}: Error finding container be10be330665131af8f733d199550342f3d278a7f6a48a574f2b911ac5ff742c: Status 404 returned error can't find the container with id be10be330665131af8f733d199550342f3d278a7f6a48a574f2b911ac5ff742c
Aug 04 18:04:47.003232 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:04:47.003206    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hg27p\" (UniqueName: \"kubernetes.io/projected/2a17f495-bbf8-4ecf-bbc5-cf9b4f32bcd7-kube-api-access-hg27p\") pod \"keda-admission-56f9798bb4-pttmg\" (UID: \"2a17f495-bbf8-4ecf-bbc5-cf9b4f32bcd7\") " pod="openshift-keda/keda-admission-56f9798bb4-pttmg"
Aug 04 18:04:47.003322 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:04:47.003241    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/2a17f495-bbf8-4ecf-bbc5-cf9b4f32bcd7-certificates\") pod \"keda-admission-56f9798bb4-pttmg\" (UID: \"2a17f495-bbf8-4ecf-bbc5-cf9b4f32bcd7\") " pod="openshift-keda/keda-admission-56f9798bb4-pttmg"
Aug 04 18:04:47.006594 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:04:47.006575    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-admission-webhooks-certs\""
Aug 04 18:04:47.012163 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:04:47.012141    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hg27p\" (UniqueName: \"kubernetes.io/projected/2a17f495-bbf8-4ecf-bbc5-cf9b4f32bcd7-kube-api-access-hg27p\") pod \"keda-admission-56f9798bb4-pttmg\" (UID: \"2a17f495-bbf8-4ecf-bbc5-cf9b4f32bcd7\") " pod="openshift-keda/keda-admission-56f9798bb4-pttmg"
Aug 04 18:04:47.015825 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:04:47.015807    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/2a17f495-bbf8-4ecf-bbc5-cf9b4f32bcd7-certificates\") pod \"keda-admission-56f9798bb4-pttmg\" (UID: \"2a17f495-bbf8-4ecf-bbc5-cf9b4f32bcd7\") " pod="openshift-keda/keda-admission-56f9798bb4-pttmg"
Aug 04 18:04:47.103036 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:04:47.102976    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-56f9798bb4-pttmg"
Aug 04 18:04:47.264386 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:04:47.264364    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-56f9798bb4-pttmg"]
Aug 04 18:04:47.266162 ip-10-0-135-124 kubenswrapper[2583]: W0804 18:04:47.266141    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a17f495_bbf8_4ecf_bbc5_cf9b4f32bcd7.slice/crio-479c8f62913e9ebf4321ba503c9ea7a28b79f22670ca9c63470d724f9fe2460c WatchSource:0}: Error finding container 479c8f62913e9ebf4321ba503c9ea7a28b79f22670ca9c63470d724f9fe2460c: Status 404 returned error can't find the container with id 479c8f62913e9ebf4321ba503c9ea7a28b79f22670ca9c63470d724f9fe2460c
Aug 04 18:04:47.414249 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:04:47.414176    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-77f648d694-fvmgq" event={"ID":"e35cd866-af62-4a66-bb6d-7ff46fa93439","Type":"ContainerStarted","Data":"400a5e846bc454eb06d7b530043e0d12b62dec4b5f08bf1d14d9e563665688f3"}
Aug 04 18:04:47.415317 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:04:47.415288    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-55557df8c-pjbv9" event={"ID":"0121a4cc-ab1c-4949-8c5b-b205ec974874","Type":"ContainerStarted","Data":"be10be330665131af8f733d199550342f3d278a7f6a48a574f2b911ac5ff742c"}
Aug 04 18:04:47.416266 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:04:47.416245    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-56f9798bb4-pttmg" event={"ID":"2a17f495-bbf8-4ecf-bbc5-cf9b4f32bcd7","Type":"ContainerStarted","Data":"479c8f62913e9ebf4321ba503c9ea7a28b79f22670ca9c63470d724f9fe2460c"}
Aug 04 18:04:51.436979 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:04:51.436942    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-56f9798bb4-pttmg" event={"ID":"2a17f495-bbf8-4ecf-bbc5-cf9b4f32bcd7","Type":"ContainerStarted","Data":"643783e6f3bacd3ea46a42bfe8b366276775078ef38946fdf4dd8b1e7630f769"}
Aug 04 18:04:51.437611 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:04:51.437448    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-admission-56f9798bb4-pttmg"
Aug 04 18:04:51.439348 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:04:51.439285    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-77f648d694-fvmgq" event={"ID":"e35cd866-af62-4a66-bb6d-7ff46fa93439","Type":"ContainerStarted","Data":"33b53fbb2df5453d7e17e7403eacd43c22502e8cc8f44376894dc6b67b6f70ca"}
Aug 04 18:04:51.439348 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:04:51.439339    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-operator-77f648d694-fvmgq"
Aug 04 18:04:51.441019 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:04:51.440988    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-55557df8c-pjbv9" event={"ID":"0121a4cc-ab1c-4949-8c5b-b205ec974874","Type":"ContainerStarted","Data":"c643770f2407ce19839bb4324ee79657a382d9047ccc675d1f7c92f46a8d572f"}
Aug 04 18:04:51.441153 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:04:51.441120    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-metrics-apiserver-55557df8c-pjbv9"
Aug 04 18:04:51.456213 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:04:51.456168    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-admission-56f9798bb4-pttmg" podStartSLOduration=1.401462138 podStartE2EDuration="5.456153211s" totalImagesPullingTime="4.054691073s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:04:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:04:51.455043195 +0000 UTC m=+305.036261773" watchObservedRunningTime="2026-08-04 18:04:51.456153211 +0000 UTC m=+305.037371767"
Aug 04 18:04:51.472766 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:04:51.472720    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-operator-77f648d694-fvmgq" podStartSLOduration=0.926343169 podStartE2EDuration="5.472702398s" totalImagesPullingTime="4.546359229s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:04:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:04:51.471303999 +0000 UTC m=+305.052522577" watchObservedRunningTime="2026-08-04 18:04:51.472702398 +0000 UTC m=+305.053920974"
Aug 04 18:04:51.490968 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:04:51.490926    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-metrics-apiserver-55557df8c-pjbv9" podStartSLOduration=1.1426061459999999 podStartE2EDuration="5.490912275s" totalImagesPullingTime="4.348306129s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:04:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:04:51.490134729 +0000 UTC m=+305.071353321" watchObservedRunningTime="2026-08-04 18:04:51.490912275 +0000 UTC m=+305.072130872"
Aug 04 18:05:02.449265 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:05:02.449234    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-metrics-apiserver-55557df8c-pjbv9"
Aug 04 18:05:07.418439 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:05:07.418410    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-4jp2h"
Aug 04 18:05:12.447183 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:05:12.447151    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-operator-77f648d694-fvmgq"
Aug 04 18:05:13.451512 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:05:13.451482    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-admission-56f9798bb4-pttmg"
Aug 04 18:05:41.568486 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:05:41.568445    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540l4snp"]
Aug 04 18:05:41.570714 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:05:41.570696    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540l4snp"
Aug 04 18:05:41.573699 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:05:41.573676    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Aug 04 18:05:41.574762 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:05:41.574739    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-wf5qp\""
Aug 04 18:05:41.574854 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:05:41.574765    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Aug 04 18:05:41.584579 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:05:41.584558    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540l4snp"]
Aug 04 18:05:41.701599 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:05:41.701565    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2ccfcf87-c5a8-457e-a02a-bf7a64617e68-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540l4snp\" (UID: \"2ccfcf87-c5a8-457e-a02a-bf7a64617e68\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540l4snp"
Aug 04 18:05:41.701754 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:05:41.701608    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mljzf\" (UniqueName: \"kubernetes.io/projected/2ccfcf87-c5a8-457e-a02a-bf7a64617e68-kube-api-access-mljzf\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540l4snp\" (UID: \"2ccfcf87-c5a8-457e-a02a-bf7a64617e68\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540l4snp"
Aug 04 18:05:41.701754 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:05:41.701647    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2ccfcf87-c5a8-457e-a02a-bf7a64617e68-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540l4snp\" (UID: \"2ccfcf87-c5a8-457e-a02a-bf7a64617e68\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540l4snp"
Aug 04 18:05:41.802964 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:05:41.802941    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2ccfcf87-c5a8-457e-a02a-bf7a64617e68-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540l4snp\" (UID: \"2ccfcf87-c5a8-457e-a02a-bf7a64617e68\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540l4snp"
Aug 04 18:05:41.803066 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:05:41.802971    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mljzf\" (UniqueName: \"kubernetes.io/projected/2ccfcf87-c5a8-457e-a02a-bf7a64617e68-kube-api-access-mljzf\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540l4snp\" (UID: \"2ccfcf87-c5a8-457e-a02a-bf7a64617e68\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540l4snp"
Aug 04 18:05:41.803066 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:05:41.803015    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2ccfcf87-c5a8-457e-a02a-bf7a64617e68-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540l4snp\" (UID: \"2ccfcf87-c5a8-457e-a02a-bf7a64617e68\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540l4snp"
Aug 04 18:05:41.803303 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:05:41.803284    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2ccfcf87-c5a8-457e-a02a-bf7a64617e68-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540l4snp\" (UID: \"2ccfcf87-c5a8-457e-a02a-bf7a64617e68\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540l4snp"
Aug 04 18:05:41.803384 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:05:41.803360    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2ccfcf87-c5a8-457e-a02a-bf7a64617e68-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540l4snp\" (UID: \"2ccfcf87-c5a8-457e-a02a-bf7a64617e68\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540l4snp"
Aug 04 18:05:41.812712 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:05:41.812689    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mljzf\" (UniqueName: \"kubernetes.io/projected/2ccfcf87-c5a8-457e-a02a-bf7a64617e68-kube-api-access-mljzf\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540l4snp\" (UID: \"2ccfcf87-c5a8-457e-a02a-bf7a64617e68\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540l4snp"
Aug 04 18:05:41.880201 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:05:41.880149    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540l4snp"
Aug 04 18:05:42.006317 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:05:42.006292    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540l4snp"]
Aug 04 18:05:42.008817 ip-10-0-135-124 kubenswrapper[2583]: W0804 18:05:42.008787    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ccfcf87_c5a8_457e_a02a_bf7a64617e68.slice/crio-e3f0c3493f3a162e154adb474815e155445d580fd3db972666f67aad14055593 WatchSource:0}: Error finding container e3f0c3493f3a162e154adb474815e155445d580fd3db972666f67aad14055593: Status 404 returned error can't find the container with id e3f0c3493f3a162e154adb474815e155445d580fd3db972666f67aad14055593
Aug 04 18:05:42.655068 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:05:42.655035    2583 generic.go:356] "Generic (PLEG): container finished" podID="2ccfcf87-c5a8-457e-a02a-bf7a64617e68" containerID="3b792d39735028a3692c4189f2aae0324e64d66678bed7c73288a86582361d1d" exitCode=0
Aug 04 18:05:42.655441 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:05:42.655126    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540l4snp" event={"ID":"2ccfcf87-c5a8-457e-a02a-bf7a64617e68","Type":"ContainerDied","Data":"3b792d39735028a3692c4189f2aae0324e64d66678bed7c73288a86582361d1d"}
Aug 04 18:05:42.655441 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:05:42.655178    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540l4snp" event={"ID":"2ccfcf87-c5a8-457e-a02a-bf7a64617e68","Type":"ContainerStarted","Data":"e3f0c3493f3a162e154adb474815e155445d580fd3db972666f67aad14055593"}
Aug 04 18:05:42.656016 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:05:42.656000    2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Aug 04 18:05:45.671443 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:05:45.671406    2583 generic.go:356] "Generic (PLEG): container finished" podID="2ccfcf87-c5a8-457e-a02a-bf7a64617e68" containerID="01f5400674b2eec9c13865b0ed60a124875bf9e0d6cae996ce28840a649348b9" exitCode=0
Aug 04 18:05:45.671836 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:05:45.671503    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540l4snp" event={"ID":"2ccfcf87-c5a8-457e-a02a-bf7a64617e68","Type":"ContainerDied","Data":"01f5400674b2eec9c13865b0ed60a124875bf9e0d6cae996ce28840a649348b9"}
Aug 04 18:05:46.677988 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:05:46.677960    2583 generic.go:356] "Generic (PLEG): container finished" podID="2ccfcf87-c5a8-457e-a02a-bf7a64617e68" containerID="32bb983f19b618e4bf8d543b9fe116ee66ec616f31153dabbf2159c575244347" exitCode=0
Aug 04 18:05:46.678357 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:05:46.678053    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540l4snp" event={"ID":"2ccfcf87-c5a8-457e-a02a-bf7a64617e68","Type":"ContainerDied","Data":"32bb983f19b618e4bf8d543b9fe116ee66ec616f31153dabbf2159c575244347"}
Aug 04 18:05:47.812650 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:05:47.812615    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540l4snp"
Aug 04 18:05:47.956046 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:05:47.955972    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2ccfcf87-c5a8-457e-a02a-bf7a64617e68-bundle\") pod \"2ccfcf87-c5a8-457e-a02a-bf7a64617e68\" (UID: \"2ccfcf87-c5a8-457e-a02a-bf7a64617e68\") "
Aug 04 18:05:47.956171 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:05:47.956060    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2ccfcf87-c5a8-457e-a02a-bf7a64617e68-util\") pod \"2ccfcf87-c5a8-457e-a02a-bf7a64617e68\" (UID: \"2ccfcf87-c5a8-457e-a02a-bf7a64617e68\") "
Aug 04 18:05:47.956171 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:05:47.956097    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mljzf\" (UniqueName: \"kubernetes.io/projected/2ccfcf87-c5a8-457e-a02a-bf7a64617e68-kube-api-access-mljzf\") pod \"2ccfcf87-c5a8-457e-a02a-bf7a64617e68\" (UID: \"2ccfcf87-c5a8-457e-a02a-bf7a64617e68\") "
Aug 04 18:05:47.956767 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:05:47.956741    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ccfcf87-c5a8-457e-a02a-bf7a64617e68-bundle" (OuterVolumeSpecName: "bundle") pod "2ccfcf87-c5a8-457e-a02a-bf7a64617e68" (UID: "2ccfcf87-c5a8-457e-a02a-bf7a64617e68"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:05:47.958108 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:05:47.958085    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ccfcf87-c5a8-457e-a02a-bf7a64617e68-kube-api-access-mljzf" (OuterVolumeSpecName: "kube-api-access-mljzf") pod "2ccfcf87-c5a8-457e-a02a-bf7a64617e68" (UID: "2ccfcf87-c5a8-457e-a02a-bf7a64617e68"). InnerVolumeSpecName "kube-api-access-mljzf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:05:47.963078 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:05:47.963055    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ccfcf87-c5a8-457e-a02a-bf7a64617e68-util" (OuterVolumeSpecName: "util") pod "2ccfcf87-c5a8-457e-a02a-bf7a64617e68" (UID: "2ccfcf87-c5a8-457e-a02a-bf7a64617e68"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:05:48.056771 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:05:48.056745    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mljzf\" (UniqueName: \"kubernetes.io/projected/2ccfcf87-c5a8-457e-a02a-bf7a64617e68-kube-api-access-mljzf\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:05:48.056771 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:05:48.056768    2583 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2ccfcf87-c5a8-457e-a02a-bf7a64617e68-bundle\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:05:48.056893 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:05:48.056779    2583 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2ccfcf87-c5a8-457e-a02a-bf7a64617e68-util\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:05:48.688359 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:05:48.688335    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540l4snp"
Aug 04 18:05:48.688359 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:05:48.688343    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540l4snp" event={"ID":"2ccfcf87-c5a8-457e-a02a-bf7a64617e68","Type":"ContainerDied","Data":"e3f0c3493f3a162e154adb474815e155445d580fd3db972666f67aad14055593"}
Aug 04 18:05:48.688594 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:05:48.688385    2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e3f0c3493f3a162e154adb474815e155445d580fd3db972666f67aad14055593"
Aug 04 18:05:54.460487 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:05:54.460438    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-cwd4h"]
Aug 04 18:05:54.460857 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:05:54.460739    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ccfcf87-c5a8-457e-a02a-bf7a64617e68" containerName="pull"
Aug 04 18:05:54.460857 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:05:54.460750    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ccfcf87-c5a8-457e-a02a-bf7a64617e68" containerName="pull"
Aug 04 18:05:54.460857 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:05:54.460766    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ccfcf87-c5a8-457e-a02a-bf7a64617e68" containerName="extract"
Aug 04 18:05:54.460857 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:05:54.460771    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ccfcf87-c5a8-457e-a02a-bf7a64617e68" containerName="extract"
Aug 04 18:05:54.460857 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:05:54.460786    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ccfcf87-c5a8-457e-a02a-bf7a64617e68" containerName="util"
Aug 04 18:05:54.460857 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:05:54.460792    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ccfcf87-c5a8-457e-a02a-bf7a64617e68" containerName="util"
Aug 04 18:05:54.460857 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:05:54.460842    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="2ccfcf87-c5a8-457e-a02a-bf7a64617e68" containerName="extract"
Aug 04 18:05:54.464522 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:05:54.464505    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-cwd4h"
Aug 04 18:05:54.468532 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:05:54.468511    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Aug 04 18:05:54.468683 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:05:54.468644    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-mph7p\""
Aug 04 18:05:54.468736 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:05:54.468727    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Aug 04 18:05:54.486955 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:05:54.486920    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-cwd4h"]
Aug 04 18:05:54.602934 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:05:54.602909    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fd84b110-b2f2-4187-b559-c5056cf57fca-tmp\") pod \"cert-manager-operator-controller-manager-684f79c9b9-cwd4h\" (UID: \"fd84b110-b2f2-4187-b559-c5056cf57fca\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-cwd4h"
Aug 04 18:05:54.603056 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:05:54.602950    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnwxn\" (UniqueName: \"kubernetes.io/projected/fd84b110-b2f2-4187-b559-c5056cf57fca-kube-api-access-fnwxn\") pod \"cert-manager-operator-controller-manager-684f79c9b9-cwd4h\" (UID: \"fd84b110-b2f2-4187-b559-c5056cf57fca\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-cwd4h"
Aug 04 18:05:54.703385 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:05:54.703355    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnwxn\" (UniqueName: \"kubernetes.io/projected/fd84b110-b2f2-4187-b559-c5056cf57fca-kube-api-access-fnwxn\") pod \"cert-manager-operator-controller-manager-684f79c9b9-cwd4h\" (UID: \"fd84b110-b2f2-4187-b559-c5056cf57fca\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-cwd4h"
Aug 04 18:05:54.703515 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:05:54.703431    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fd84b110-b2f2-4187-b559-c5056cf57fca-tmp\") pod \"cert-manager-operator-controller-manager-684f79c9b9-cwd4h\" (UID: \"fd84b110-b2f2-4187-b559-c5056cf57fca\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-cwd4h"
Aug 04 18:05:54.703811 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:05:54.703791    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fd84b110-b2f2-4187-b559-c5056cf57fca-tmp\") pod \"cert-manager-operator-controller-manager-684f79c9b9-cwd4h\" (UID: \"fd84b110-b2f2-4187-b559-c5056cf57fca\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-cwd4h"
Aug 04 18:05:54.714967 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:05:54.714907    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnwxn\" (UniqueName: \"kubernetes.io/projected/fd84b110-b2f2-4187-b559-c5056cf57fca-kube-api-access-fnwxn\") pod \"cert-manager-operator-controller-manager-684f79c9b9-cwd4h\" (UID: \"fd84b110-b2f2-4187-b559-c5056cf57fca\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-cwd4h"
Aug 04 18:05:54.773478 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:05:54.773450    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-cwd4h"
Aug 04 18:05:54.911115 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:05:54.911092    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-cwd4h"]
Aug 04 18:05:54.914000 ip-10-0-135-124 kubenswrapper[2583]: W0804 18:05:54.913966    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd84b110_b2f2_4187_b559_c5056cf57fca.slice/crio-6a0f64bb5e2caf8eb41cbf089cad943bd431373d55071855a2a92020dd995251 WatchSource:0}: Error finding container 6a0f64bb5e2caf8eb41cbf089cad943bd431373d55071855a2a92020dd995251: Status 404 returned error can't find the container with id 6a0f64bb5e2caf8eb41cbf089cad943bd431373d55071855a2a92020dd995251
Aug 04 18:05:55.721984 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:05:55.721952    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-cwd4h" event={"ID":"fd84b110-b2f2-4187-b559-c5056cf57fca","Type":"ContainerStarted","Data":"6a0f64bb5e2caf8eb41cbf089cad943bd431373d55071855a2a92020dd995251"}
Aug 04 18:05:57.733877 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:05:57.733834    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-cwd4h" event={"ID":"fd84b110-b2f2-4187-b559-c5056cf57fca","Type":"ContainerStarted","Data":"2506faf6e6c77b5bfab421787a38830f4a9b3b4f913310abedbaea8ce3397553"}
Aug 04 18:05:57.756780 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:05:57.756734    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-cwd4h" podStartSLOduration=1.9440334240000001 podStartE2EDuration="3.75672003s" totalImagesPullingTime="1.812686606s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:05:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:05:57.753904637 +0000 UTC m=+371.335123215" watchObservedRunningTime="2026-08-04 18:05:57.75672003 +0000 UTC m=+371.337938607"
Aug 04 18:06:03.720333 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:06:03.720298    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ff2jl4"]
Aug 04 18:06:03.723856 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:06:03.723838    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ff2jl4"
Aug 04 18:06:03.726735 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:06:03.726714    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Aug 04 18:06:03.727919 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:06:03.727903    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-wf5qp\""
Aug 04 18:06:03.728046 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:06:03.727892    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Aug 04 18:06:03.734666 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:06:03.734644    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ff2jl4"]
Aug 04 18:06:03.874222 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:06:03.874194    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/02c82746-3a00-487e-9a0c-4cf2f962791f-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ff2jl4\" (UID: \"02c82746-3a00-487e-9a0c-4cf2f962791f\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ff2jl4"
Aug 04 18:06:03.874353 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:06:03.874276    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bs8n9\" (UniqueName: \"kubernetes.io/projected/02c82746-3a00-487e-9a0c-4cf2f962791f-kube-api-access-bs8n9\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ff2jl4\" (UID: \"02c82746-3a00-487e-9a0c-4cf2f962791f\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ff2jl4"
Aug 04 18:06:03.874353 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:06:03.874300    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/02c82746-3a00-487e-9a0c-4cf2f962791f-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ff2jl4\" (UID: \"02c82746-3a00-487e-9a0c-4cf2f962791f\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ff2jl4"
Aug 04 18:06:03.975265 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:06:03.975205    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bs8n9\" (UniqueName: \"kubernetes.io/projected/02c82746-3a00-487e-9a0c-4cf2f962791f-kube-api-access-bs8n9\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ff2jl4\" (UID: \"02c82746-3a00-487e-9a0c-4cf2f962791f\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ff2jl4"
Aug 04 18:06:03.975265 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:06:03.975234    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/02c82746-3a00-487e-9a0c-4cf2f962791f-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ff2jl4\" (UID: \"02c82746-3a00-487e-9a0c-4cf2f962791f\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ff2jl4"
Aug 04 18:06:03.975265 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:06:03.975261    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/02c82746-3a00-487e-9a0c-4cf2f962791f-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ff2jl4\" (UID: \"02c82746-3a00-487e-9a0c-4cf2f962791f\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ff2jl4"
Aug 04 18:06:03.975586 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:06:03.975567    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/02c82746-3a00-487e-9a0c-4cf2f962791f-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ff2jl4\" (UID: \"02c82746-3a00-487e-9a0c-4cf2f962791f\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ff2jl4"
Aug 04 18:06:03.975628 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:06:03.975597    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/02c82746-3a00-487e-9a0c-4cf2f962791f-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ff2jl4\" (UID: \"02c82746-3a00-487e-9a0c-4cf2f962791f\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ff2jl4"
Aug 04 18:06:03.984283 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:06:03.984257    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bs8n9\" (UniqueName: \"kubernetes.io/projected/02c82746-3a00-487e-9a0c-4cf2f962791f-kube-api-access-bs8n9\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ff2jl4\" (UID: \"02c82746-3a00-487e-9a0c-4cf2f962791f\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ff2jl4"
Aug 04 18:06:04.032743 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:06:04.032719    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ff2jl4"
Aug 04 18:06:04.166784 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:06:04.166761    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ff2jl4"]
Aug 04 18:06:04.169410 ip-10-0-135-124 kubenswrapper[2583]: W0804 18:06:04.169380    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02c82746_3a00_487e_9a0c_4cf2f962791f.slice/crio-fedc182c57d4e1c2bc919151779a45e2f1866e31a679bef4bf51c2b21c6ff3e4 WatchSource:0}: Error finding container fedc182c57d4e1c2bc919151779a45e2f1866e31a679bef4bf51c2b21c6ff3e4: Status 404 returned error can't find the container with id fedc182c57d4e1c2bc919151779a45e2f1866e31a679bef4bf51c2b21c6ff3e4
Aug 04 18:06:04.768864 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:06:04.768826    2583 generic.go:356] "Generic (PLEG): container finished" podID="02c82746-3a00-487e-9a0c-4cf2f962791f" containerID="accf58f65dd0d12e97e596e242c99ef21aab8c6c10f8216935570b5d47b9f800" exitCode=0
Aug 04 18:06:04.769233 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:06:04.768908    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ff2jl4" event={"ID":"02c82746-3a00-487e-9a0c-4cf2f962791f","Type":"ContainerDied","Data":"accf58f65dd0d12e97e596e242c99ef21aab8c6c10f8216935570b5d47b9f800"}
Aug 04 18:06:04.769233 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:06:04.768957    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ff2jl4" event={"ID":"02c82746-3a00-487e-9a0c-4cf2f962791f","Type":"ContainerStarted","Data":"fedc182c57d4e1c2bc919151779a45e2f1866e31a679bef4bf51c2b21c6ff3e4"}
Aug 04 18:06:07.784133 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:06:07.784102    2583 generic.go:356] "Generic (PLEG): container finished" podID="02c82746-3a00-487e-9a0c-4cf2f962791f" containerID="24f046fe7eb5d0b06192676f45dd8f045a2771021230a05b20ff63e104db7571" exitCode=0
Aug 04 18:06:07.784619 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:06:07.784190    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ff2jl4" event={"ID":"02c82746-3a00-487e-9a0c-4cf2f962791f","Type":"ContainerDied","Data":"24f046fe7eb5d0b06192676f45dd8f045a2771021230a05b20ff63e104db7571"}
Aug 04 18:06:08.794569 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:06:08.792306    2583 generic.go:356] "Generic (PLEG): container finished" podID="02c82746-3a00-487e-9a0c-4cf2f962791f" containerID="4c9790981a9672721911b1299f6f99223e64b1ef69e1a809c096743b771c586b" exitCode=0
Aug 04 18:06:08.794569 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:06:08.792462    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ff2jl4" event={"ID":"02c82746-3a00-487e-9a0c-4cf2f962791f","Type":"ContainerDied","Data":"4c9790981a9672721911b1299f6f99223e64b1ef69e1a809c096743b771c586b"}
Aug 04 18:06:09.923737 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:06:09.923712    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ff2jl4"
Aug 04 18:06:10.022789 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:06:10.022764    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/02c82746-3a00-487e-9a0c-4cf2f962791f-bundle\") pod \"02c82746-3a00-487e-9a0c-4cf2f962791f\" (UID: \"02c82746-3a00-487e-9a0c-4cf2f962791f\") "
Aug 04 18:06:10.022939 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:06:10.022826    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/02c82746-3a00-487e-9a0c-4cf2f962791f-util\") pod \"02c82746-3a00-487e-9a0c-4cf2f962791f\" (UID: \"02c82746-3a00-487e-9a0c-4cf2f962791f\") "
Aug 04 18:06:10.022939 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:06:10.022843    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bs8n9\" (UniqueName: \"kubernetes.io/projected/02c82746-3a00-487e-9a0c-4cf2f962791f-kube-api-access-bs8n9\") pod \"02c82746-3a00-487e-9a0c-4cf2f962791f\" (UID: \"02c82746-3a00-487e-9a0c-4cf2f962791f\") "
Aug 04 18:06:10.023101 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:06:10.023079    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02c82746-3a00-487e-9a0c-4cf2f962791f-bundle" (OuterVolumeSpecName: "bundle") pod "02c82746-3a00-487e-9a0c-4cf2f962791f" (UID: "02c82746-3a00-487e-9a0c-4cf2f962791f"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:06:10.024894 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:06:10.024868    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02c82746-3a00-487e-9a0c-4cf2f962791f-kube-api-access-bs8n9" (OuterVolumeSpecName: "kube-api-access-bs8n9") pod "02c82746-3a00-487e-9a0c-4cf2f962791f" (UID: "02c82746-3a00-487e-9a0c-4cf2f962791f"). InnerVolumeSpecName "kube-api-access-bs8n9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:06:10.027427 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:06:10.027408    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02c82746-3a00-487e-9a0c-4cf2f962791f-util" (OuterVolumeSpecName: "util") pod "02c82746-3a00-487e-9a0c-4cf2f962791f" (UID: "02c82746-3a00-487e-9a0c-4cf2f962791f"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:06:10.123693 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:06:10.123637    2583 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/02c82746-3a00-487e-9a0c-4cf2f962791f-bundle\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:06:10.123693 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:06:10.123657    2583 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/02c82746-3a00-487e-9a0c-4cf2f962791f-util\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:06:10.123693 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:06:10.123666    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bs8n9\" (UniqueName: \"kubernetes.io/projected/02c82746-3a00-487e-9a0c-4cf2f962791f-kube-api-access-bs8n9\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:06:10.803020 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:06:10.802982    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ff2jl4" event={"ID":"02c82746-3a00-487e-9a0c-4cf2f962791f","Type":"ContainerDied","Data":"fedc182c57d4e1c2bc919151779a45e2f1866e31a679bef4bf51c2b21c6ff3e4"}
Aug 04 18:06:10.803177 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:06:10.803031    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ff2jl4"
Aug 04 18:06:10.803177 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:06:10.803037    2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fedc182c57d4e1c2bc919151779a45e2f1866e31a679bef4bf51c2b21c6ff3e4"
Aug 04 18:06:31.098123 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:06:31.098089    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lvpcs"]
Aug 04 18:06:31.098647 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:06:31.098529    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02c82746-3a00-487e-9a0c-4cf2f962791f" containerName="extract"
Aug 04 18:06:31.098647 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:06:31.098547    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="02c82746-3a00-487e-9a0c-4cf2f962791f" containerName="extract"
Aug 04 18:06:31.098647 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:06:31.098568    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02c82746-3a00-487e-9a0c-4cf2f962791f" containerName="pull"
Aug 04 18:06:31.098647 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:06:31.098576    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="02c82746-3a00-487e-9a0c-4cf2f962791f" containerName="pull"
Aug 04 18:06:31.098647 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:06:31.098596    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02c82746-3a00-487e-9a0c-4cf2f962791f" containerName="util"
Aug 04 18:06:31.098647 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:06:31.098605    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="02c82746-3a00-487e-9a0c-4cf2f962791f" containerName="util"
Aug 04 18:06:31.098932 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:06:31.098691    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="02c82746-3a00-487e-9a0c-4cf2f962791f" containerName="extract"
Aug 04 18:06:31.103511 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:06:31.103491    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lvpcs"
Aug 04 18:06:31.106181 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:06:31.106160    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Aug 04 18:06:31.106295 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:06:31.106174    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-2rpv5\""
Aug 04 18:06:31.106295 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:06:31.106269    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Aug 04 18:06:31.106295 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:06:31.106278    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Aug 04 18:06:31.106446 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:06:31.106292    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Aug 04 18:06:31.113543 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:06:31.113525    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lvpcs"]
Aug 04 18:06:31.183539 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:06:31.183513    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/8e649f5f-ed69-42cf-b451-4755a5243ae7-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-lvpcs\" (UID: \"8e649f5f-ed69-42cf-b451-4755a5243ae7\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lvpcs"
Aug 04 18:06:31.183687 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:06:31.183546    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e649f5f-ed69-42cf-b451-4755a5243ae7-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-lvpcs\" (UID: \"8e649f5f-ed69-42cf-b451-4755a5243ae7\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lvpcs"
Aug 04 18:06:31.183687 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:06:31.183564    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6l6rj\" (UniqueName: \"kubernetes.io/projected/8e649f5f-ed69-42cf-b451-4755a5243ae7-kube-api-access-6l6rj\") pod \"istiod-openshift-gateway-7b7995d5cd-lvpcs\" (UID: \"8e649f5f-ed69-42cf-b451-4755a5243ae7\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lvpcs"
Aug 04 18:06:31.183687 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:06:31.183654    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8e649f5f-ed69-42cf-b451-4755a5243ae7-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-lvpcs\" (UID: \"8e649f5f-ed69-42cf-b451-4755a5243ae7\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lvpcs"
Aug 04 18:06:31.183819 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:06:31.183705    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/8e649f5f-ed69-42cf-b451-4755a5243ae7-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-lvpcs\" (UID: \"8e649f5f-ed69-42cf-b451-4755a5243ae7\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lvpcs"
Aug 04 18:06:31.183819 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:06:31.183781    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/8e649f5f-ed69-42cf-b451-4755a5243ae7-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-lvpcs\" (UID: \"8e649f5f-ed69-42cf-b451-4755a5243ae7\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lvpcs"
Aug 04 18:06:31.183819 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:06:31.183810    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e649f5f-ed69-42cf-b451-4755a5243ae7-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-lvpcs\" (UID: \"8e649f5f-ed69-42cf-b451-4755a5243ae7\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lvpcs"
Aug 04 18:06:31.284604 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:06:31.284577    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/8e649f5f-ed69-42cf-b451-4755a5243ae7-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-lvpcs\" (UID: \"8e649f5f-ed69-42cf-b451-4755a5243ae7\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lvpcs"
Aug 04 18:06:31.284712 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:06:31.284611    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e649f5f-ed69-42cf-b451-4755a5243ae7-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-lvpcs\" (UID: \"8e649f5f-ed69-42cf-b451-4755a5243ae7\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lvpcs"
Aug 04 18:06:31.284712 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:06:31.284628    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6l6rj\" (UniqueName: \"kubernetes.io/projected/8e649f5f-ed69-42cf-b451-4755a5243ae7-kube-api-access-6l6rj\") pod \"istiod-openshift-gateway-7b7995d5cd-lvpcs\" (UID: \"8e649f5f-ed69-42cf-b451-4755a5243ae7\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lvpcs"
Aug 04 18:06:31.284712 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:06:31.284690    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8e649f5f-ed69-42cf-b451-4755a5243ae7-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-lvpcs\" (UID: \"8e649f5f-ed69-42cf-b451-4755a5243ae7\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lvpcs"
Aug 04 18:06:31.284860 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:06:31.284714    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/8e649f5f-ed69-42cf-b451-4755a5243ae7-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-lvpcs\" (UID: \"8e649f5f-ed69-42cf-b451-4755a5243ae7\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lvpcs"
Aug 04 18:06:31.284860 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:06:31.284756    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/8e649f5f-ed69-42cf-b451-4755a5243ae7-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-lvpcs\" (UID: \"8e649f5f-ed69-42cf-b451-4755a5243ae7\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lvpcs"
Aug 04 18:06:31.284860 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:06:31.284789    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e649f5f-ed69-42cf-b451-4755a5243ae7-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-lvpcs\" (UID: \"8e649f5f-ed69-42cf-b451-4755a5243ae7\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lvpcs"
Aug 04 18:06:31.285305 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:06:31.285282    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/8e649f5f-ed69-42cf-b451-4755a5243ae7-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-lvpcs\" (UID: \"8e649f5f-ed69-42cf-b451-4755a5243ae7\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lvpcs"
Aug 04 18:06:31.287043 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:06:31.287018    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/8e649f5f-ed69-42cf-b451-4755a5243ae7-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-lvpcs\" (UID: \"8e649f5f-ed69-42cf-b451-4755a5243ae7\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lvpcs"
Aug 04 18:06:31.287155 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:06:31.287038    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e649f5f-ed69-42cf-b451-4755a5243ae7-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-lvpcs\" (UID: \"8e649f5f-ed69-42cf-b451-4755a5243ae7\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lvpcs"
Aug 04 18:06:31.287155 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:06:31.287068    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8e649f5f-ed69-42cf-b451-4755a5243ae7-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-lvpcs\" (UID: \"8e649f5f-ed69-42cf-b451-4755a5243ae7\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lvpcs"
Aug 04 18:06:31.287267 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:06:31.287159    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/8e649f5f-ed69-42cf-b451-4755a5243ae7-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-lvpcs\" (UID: \"8e649f5f-ed69-42cf-b451-4755a5243ae7\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lvpcs"
Aug 04 18:06:31.295004 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:06:31.294977    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6l6rj\" (UniqueName: \"kubernetes.io/projected/8e649f5f-ed69-42cf-b451-4755a5243ae7-kube-api-access-6l6rj\") pod \"istiod-openshift-gateway-7b7995d5cd-lvpcs\" (UID: \"8e649f5f-ed69-42cf-b451-4755a5243ae7\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lvpcs"
Aug 04 18:06:31.295004 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:06:31.294997    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e649f5f-ed69-42cf-b451-4755a5243ae7-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-lvpcs\" (UID: \"8e649f5f-ed69-42cf-b451-4755a5243ae7\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lvpcs"
Aug 04 18:06:31.413426 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:06:31.413372    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lvpcs"
Aug 04 18:06:31.543993 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:06:31.543932    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lvpcs"]
Aug 04 18:06:31.546200 ip-10-0-135-124 kubenswrapper[2583]: W0804 18:06:31.546165    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e649f5f_ed69_42cf_b451_4755a5243ae7.slice/crio-d16b6a83837cfda04265a09c3321fbd5aac11ae264ac5eee17c16307e91406fb WatchSource:0}: Error finding container d16b6a83837cfda04265a09c3321fbd5aac11ae264ac5eee17c16307e91406fb: Status 404 returned error can't find the container with id d16b6a83837cfda04265a09c3321fbd5aac11ae264ac5eee17c16307e91406fb
Aug 04 18:06:31.899067 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:06:31.899034    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lvpcs" event={"ID":"8e649f5f-ed69-42cf-b451-4755a5243ae7","Type":"ContainerStarted","Data":"d16b6a83837cfda04265a09c3321fbd5aac11ae264ac5eee17c16307e91406fb"}
Aug 04 18:06:34.032511 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:06:34.032444    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Aug 04 18:06:34.032749 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:06:34.032542    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Aug 04 18:06:34.916523 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:06:34.916457    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lvpcs" event={"ID":"8e649f5f-ed69-42cf-b451-4755a5243ae7","Type":"ContainerStarted","Data":"d0b8c61d9624e617d3986ba16e2cc903ed83f1a2617d0030d2637ae7674e5417"}
Aug 04 18:06:34.916687 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:06:34.916604    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lvpcs"
Aug 04 18:06:34.942067 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:06:34.942012    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lvpcs" podStartSLOduration=1.457819546 podStartE2EDuration="3.94199064s" totalImagesPullingTime="2.484171094s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:06:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:06:34.939195436 +0000 UTC m=+408.520414026" watchObservedRunningTime="2026-08-04 18:06:34.94199064 +0000 UTC m=+408.523209219"
Aug 04 18:06:35.923337 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:06:35.923309    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lvpcs"
Aug 04 18:06:58.250662 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:06:58.250619    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870539ksjk"]
Aug 04 18:06:58.256901 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:06:58.256881    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870539ksjk"
Aug 04 18:06:58.259721 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:06:58.259700    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Aug 04 18:06:58.259828 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:06:58.259735    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Aug 04 18:06:58.260872 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:06:58.260848    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-wf5qp\""
Aug 04 18:06:58.263440 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:06:58.263417    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870539ksjk"]
Aug 04 18:06:58.372641 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:06:58.372604    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c29trg"]
Aug 04 18:06:58.376263 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:06:58.376248    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c29trg"
Aug 04 18:06:58.394004 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:06:58.393982    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c29trg"]
Aug 04 18:06:58.407844 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:06:58.407816    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d44b74d5-56cf-423a-8bf6-8c325024a661-bundle\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870539ksjk\" (UID: \"d44b74d5-56cf-423a-8bf6-8c325024a661\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870539ksjk"
Aug 04 18:06:58.407968 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:06:58.407881    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kckxx\" (UniqueName: \"kubernetes.io/projected/d44b74d5-56cf-423a-8bf6-8c325024a661-kube-api-access-kckxx\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870539ksjk\" (UID: \"d44b74d5-56cf-423a-8bf6-8c325024a661\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870539ksjk"
Aug 04 18:06:58.407968 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:06:58.407950    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d44b74d5-56cf-423a-8bf6-8c325024a661-util\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870539ksjk\" (UID: \"d44b74d5-56cf-423a-8bf6-8c325024a661\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870539ksjk"
Aug 04 18:06:58.455510 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:06:58.455481    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb6llcz"]
Aug 04 18:06:58.464867 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:06:58.464843    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb6llcz"
Aug 04 18:06:58.468211 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:06:58.468183    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb6llcz"]
Aug 04 18:06:58.509412 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:06:58.509352    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kckxx\" (UniqueName: \"kubernetes.io/projected/d44b74d5-56cf-423a-8bf6-8c325024a661-kube-api-access-kckxx\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870539ksjk\" (UID: \"d44b74d5-56cf-423a-8bf6-8c325024a661\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870539ksjk"
Aug 04 18:06:58.509412 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:06:58.509388    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c6e0e155-dddf-427c-a55c-bee6b6b0fbfd-util\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c29trg\" (UID: \"c6e0e155-dddf-427c-a55c-bee6b6b0fbfd\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c29trg"
Aug 04 18:06:58.509412 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:06:58.509406    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ql7f6\" (UniqueName: \"kubernetes.io/projected/c6e0e155-dddf-427c-a55c-bee6b6b0fbfd-kube-api-access-ql7f6\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c29trg\" (UID: \"c6e0e155-dddf-427c-a55c-bee6b6b0fbfd\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c29trg"
Aug 04 18:06:58.509664 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:06:58.509433    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c6e0e155-dddf-427c-a55c-bee6b6b0fbfd-bundle\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c29trg\" (UID: \"c6e0e155-dddf-427c-a55c-bee6b6b0fbfd\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c29trg"
Aug 04 18:06:58.509664 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:06:58.509519    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d44b74d5-56cf-423a-8bf6-8c325024a661-util\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870539ksjk\" (UID: \"d44b74d5-56cf-423a-8bf6-8c325024a661\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870539ksjk"
Aug 04 18:06:58.509664 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:06:58.509610    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d44b74d5-56cf-423a-8bf6-8c325024a661-bundle\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870539ksjk\" (UID: \"d44b74d5-56cf-423a-8bf6-8c325024a661\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870539ksjk"
Aug 04 18:06:58.509945 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:06:58.509924    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d44b74d5-56cf-423a-8bf6-8c325024a661-util\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870539ksjk\" (UID: \"d44b74d5-56cf-423a-8bf6-8c325024a661\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870539ksjk"
Aug 04 18:06:58.509945 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:06:58.509937    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d44b74d5-56cf-423a-8bf6-8c325024a661-bundle\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870539ksjk\" (UID: \"d44b74d5-56cf-423a-8bf6-8c325024a661\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870539ksjk"
Aug 04 18:06:58.518063 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:06:58.518042    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kckxx\" (UniqueName: \"kubernetes.io/projected/d44b74d5-56cf-423a-8bf6-8c325024a661-kube-api-access-kckxx\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870539ksjk\" (UID: \"d44b74d5-56cf-423a-8bf6-8c325024a661\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870539ksjk"
Aug 04 18:06:58.549233 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:06:58.549205    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0p2s68"]
Aug 04 18:06:58.553144 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:06:58.553128    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0p2s68"
Aug 04 18:06:58.561461 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:06:58.561439    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0p2s68"]
Aug 04 18:06:58.568207 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:06:58.568189    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870539ksjk"
Aug 04 18:06:58.610857 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:06:58.610811    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c6e0e155-dddf-427c-a55c-bee6b6b0fbfd-bundle\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c29trg\" (UID: \"c6e0e155-dddf-427c-a55c-bee6b6b0fbfd\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c29trg"
Aug 04 18:06:58.611029 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:06:58.610863    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bd9ea619-2bb1-42d9-a67b-3331ad969d81-bundle\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb6llcz\" (UID: \"bd9ea619-2bb1-42d9-a67b-3331ad969d81\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb6llcz"
Aug 04 18:06:58.611029 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:06:58.610996    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bd9ea619-2bb1-42d9-a67b-3331ad969d81-util\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb6llcz\" (UID: \"bd9ea619-2bb1-42d9-a67b-3331ad969d81\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb6llcz"
Aug 04 18:06:58.611121 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:06:58.611040    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c6e0e155-dddf-427c-a55c-bee6b6b0fbfd-util\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c29trg\" (UID: \"c6e0e155-dddf-427c-a55c-bee6b6b0fbfd\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c29trg"
Aug 04 18:06:58.611121 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:06:58.611069    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ql7f6\" (UniqueName: \"kubernetes.io/projected/c6e0e155-dddf-427c-a55c-bee6b6b0fbfd-kube-api-access-ql7f6\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c29trg\" (UID: \"c6e0e155-dddf-427c-a55c-bee6b6b0fbfd\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c29trg"
Aug 04 18:06:58.611121 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:06:58.611095    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qc8w2\" (UniqueName: \"kubernetes.io/projected/bd9ea619-2bb1-42d9-a67b-3331ad969d81-kube-api-access-qc8w2\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb6llcz\" (UID: \"bd9ea619-2bb1-42d9-a67b-3331ad969d81\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb6llcz"
Aug 04 18:06:58.611264 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:06:58.611252    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c6e0e155-dddf-427c-a55c-bee6b6b0fbfd-bundle\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c29trg\" (UID: \"c6e0e155-dddf-427c-a55c-bee6b6b0fbfd\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c29trg"
Aug 04 18:06:58.611411 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:06:58.611387    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c6e0e155-dddf-427c-a55c-bee6b6b0fbfd-util\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c29trg\" (UID: \"c6e0e155-dddf-427c-a55c-bee6b6b0fbfd\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c29trg"
Aug 04 18:06:58.621957 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:06:58.621229    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ql7f6\" (UniqueName: \"kubernetes.io/projected/c6e0e155-dddf-427c-a55c-bee6b6b0fbfd-kube-api-access-ql7f6\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c29trg\" (UID: \"c6e0e155-dddf-427c-a55c-bee6b6b0fbfd\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c29trg"
Aug 04 18:06:58.685035 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:06:58.684996    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c29trg"
Aug 04 18:06:58.704992 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:06:58.704969    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870539ksjk"]
Aug 04 18:06:58.707517 ip-10-0-135-124 kubenswrapper[2583]: W0804 18:06:58.707490    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd44b74d5_56cf_423a_8bf6_8c325024a661.slice/crio-259f50c458c03606e126bfa9b3e008d0e43a0fe55e6fa2bf376e4445482c9f45 WatchSource:0}: Error finding container 259f50c458c03606e126bfa9b3e008d0e43a0fe55e6fa2bf376e4445482c9f45: Status 404 returned error can't find the container with id 259f50c458c03606e126bfa9b3e008d0e43a0fe55e6fa2bf376e4445482c9f45
Aug 04 18:06:58.711724 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:06:58.711694    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmjt7\" (UniqueName: \"kubernetes.io/projected/5ad18d44-3fa9-4b61-a206-c0195cabc7c7-kube-api-access-zmjt7\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0p2s68\" (UID: \"5ad18d44-3fa9-4b61-a206-c0195cabc7c7\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0p2s68"
Aug 04 18:06:58.711848 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:06:58.711760    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bd9ea619-2bb1-42d9-a67b-3331ad969d81-util\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb6llcz\" (UID: \"bd9ea619-2bb1-42d9-a67b-3331ad969d81\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb6llcz"
Aug 04 18:06:58.711848 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:06:58.711790    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qc8w2\" (UniqueName: \"kubernetes.io/projected/bd9ea619-2bb1-42d9-a67b-3331ad969d81-kube-api-access-qc8w2\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb6llcz\" (UID: \"bd9ea619-2bb1-42d9-a67b-3331ad969d81\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb6llcz"
Aug 04 18:06:58.711848 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:06:58.711821    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5ad18d44-3fa9-4b61-a206-c0195cabc7c7-util\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0p2s68\" (UID: \"5ad18d44-3fa9-4b61-a206-c0195cabc7c7\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0p2s68"
Aug 04 18:06:58.711848 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:06:58.711841    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bd9ea619-2bb1-42d9-a67b-3331ad969d81-bundle\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb6llcz\" (UID: \"bd9ea619-2bb1-42d9-a67b-3331ad969d81\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb6llcz"
Aug 04 18:06:58.712047 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:06:58.711863    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5ad18d44-3fa9-4b61-a206-c0195cabc7c7-bundle\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0p2s68\" (UID: \"5ad18d44-3fa9-4b61-a206-c0195cabc7c7\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0p2s68"
Aug 04 18:06:58.712113 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:06:58.712095    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bd9ea619-2bb1-42d9-a67b-3331ad969d81-util\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb6llcz\" (UID: \"bd9ea619-2bb1-42d9-a67b-3331ad969d81\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb6llcz"
Aug 04 18:06:58.712168 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:06:58.712147    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bd9ea619-2bb1-42d9-a67b-3331ad969d81-bundle\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb6llcz\" (UID: \"bd9ea619-2bb1-42d9-a67b-3331ad969d81\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb6llcz"
Aug 04 18:06:58.720436 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:06:58.720405    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qc8w2\" (UniqueName: \"kubernetes.io/projected/bd9ea619-2bb1-42d9-a67b-3331ad969d81-kube-api-access-qc8w2\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb6llcz\" (UID: \"bd9ea619-2bb1-42d9-a67b-3331ad969d81\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb6llcz"
Aug 04 18:06:58.775179 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:06:58.775154    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb6llcz"
Aug 04 18:06:58.813445 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:06:58.812679    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmjt7\" (UniqueName: \"kubernetes.io/projected/5ad18d44-3fa9-4b61-a206-c0195cabc7c7-kube-api-access-zmjt7\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0p2s68\" (UID: \"5ad18d44-3fa9-4b61-a206-c0195cabc7c7\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0p2s68"
Aug 04 18:06:58.813445 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:06:58.812780    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5ad18d44-3fa9-4b61-a206-c0195cabc7c7-util\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0p2s68\" (UID: \"5ad18d44-3fa9-4b61-a206-c0195cabc7c7\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0p2s68"
Aug 04 18:06:58.813445 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:06:58.812815    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5ad18d44-3fa9-4b61-a206-c0195cabc7c7-bundle\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0p2s68\" (UID: \"5ad18d44-3fa9-4b61-a206-c0195cabc7c7\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0p2s68"
Aug 04 18:06:58.813445 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:06:58.813307    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5ad18d44-3fa9-4b61-a206-c0195cabc7c7-bundle\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0p2s68\" (UID: \"5ad18d44-3fa9-4b61-a206-c0195cabc7c7\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0p2s68"
Aug 04 18:06:58.814659 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:06:58.813859    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5ad18d44-3fa9-4b61-a206-c0195cabc7c7-util\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0p2s68\" (UID: \"5ad18d44-3fa9-4b61-a206-c0195cabc7c7\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0p2s68"
Aug 04 18:06:58.824079 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:06:58.824051    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmjt7\" (UniqueName: \"kubernetes.io/projected/5ad18d44-3fa9-4b61-a206-c0195cabc7c7-kube-api-access-zmjt7\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0p2s68\" (UID: \"5ad18d44-3fa9-4b61-a206-c0195cabc7c7\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0p2s68"
Aug 04 18:06:58.836241 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:06:58.836164    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c29trg"]
Aug 04 18:06:58.862728 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:06:58.862693    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0p2s68"
Aug 04 18:06:58.865301 ip-10-0-135-124 kubenswrapper[2583]: W0804 18:06:58.865271    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6e0e155_dddf_427c_a55c_bee6b6b0fbfd.slice/crio-9ae405446806c9d0f216c3ae6646589a4a190aaacdf33b9d80ca10651d4409fe WatchSource:0}: Error finding container 9ae405446806c9d0f216c3ae6646589a4a190aaacdf33b9d80ca10651d4409fe: Status 404 returned error can't find the container with id 9ae405446806c9d0f216c3ae6646589a4a190aaacdf33b9d80ca10651d4409fe
Aug 04 18:06:58.925047 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:06:58.925013    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb6llcz"]
Aug 04 18:06:58.928419 ip-10-0-135-124 kubenswrapper[2583]: W0804 18:06:58.928389    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd9ea619_2bb1_42d9_a67b_3331ad969d81.slice/crio-0bf4ba6ac65ab185d59058fed70408387fe97ddeabb1005d5369ce27117e8a89 WatchSource:0}: Error finding container 0bf4ba6ac65ab185d59058fed70408387fe97ddeabb1005d5369ce27117e8a89: Status 404 returned error can't find the container with id 0bf4ba6ac65ab185d59058fed70408387fe97ddeabb1005d5369ce27117e8a89
Aug 04 18:06:59.023796 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:06:59.023449    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0p2s68"]
Aug 04 18:06:59.024719 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:06:59.024694    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb6llcz" event={"ID":"bd9ea619-2bb1-42d9-a67b-3331ad969d81","Type":"ContainerStarted","Data":"00a2b36b876cf07d366fe28e3953cd7f6bc412f8cff82b9d628ee8c4b426f7b7"}
Aug 04 18:06:59.024859 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:06:59.024740    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb6llcz" event={"ID":"bd9ea619-2bb1-42d9-a67b-3331ad969d81","Type":"ContainerStarted","Data":"0bf4ba6ac65ab185d59058fed70408387fe97ddeabb1005d5369ce27117e8a89"}
Aug 04 18:06:59.026555 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:06:59.026531    2583 generic.go:356] "Generic (PLEG): container finished" podID="d44b74d5-56cf-423a-8bf6-8c325024a661" containerID="57a64b1f3cdc4c0bc31ab77229643f8a6abee8984dea992f4e1ed6b885e19762" exitCode=0
Aug 04 18:06:59.026662 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:06:59.026566    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870539ksjk" event={"ID":"d44b74d5-56cf-423a-8bf6-8c325024a661","Type":"ContainerDied","Data":"57a64b1f3cdc4c0bc31ab77229643f8a6abee8984dea992f4e1ed6b885e19762"}
Aug 04 18:06:59.026662 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:06:59.026611    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870539ksjk" event={"ID":"d44b74d5-56cf-423a-8bf6-8c325024a661","Type":"ContainerStarted","Data":"259f50c458c03606e126bfa9b3e008d0e43a0fe55e6fa2bf376e4445482c9f45"}
Aug 04 18:06:59.028626 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:06:59.028599    2583 generic.go:356] "Generic (PLEG): container finished" podID="c6e0e155-dddf-427c-a55c-bee6b6b0fbfd" containerID="23860038be556cfedc99d32abeab697f4eaaab808754dccf4eb9ef2d529001d0" exitCode=0
Aug 04 18:06:59.028741 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:06:59.028638    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c29trg" event={"ID":"c6e0e155-dddf-427c-a55c-bee6b6b0fbfd","Type":"ContainerDied","Data":"23860038be556cfedc99d32abeab697f4eaaab808754dccf4eb9ef2d529001d0"}
Aug 04 18:06:59.028741 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:06:59.028671    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c29trg" event={"ID":"c6e0e155-dddf-427c-a55c-bee6b6b0fbfd","Type":"ContainerStarted","Data":"9ae405446806c9d0f216c3ae6646589a4a190aaacdf33b9d80ca10651d4409fe"}
Aug 04 18:06:59.087336 ip-10-0-135-124 kubenswrapper[2583]: W0804 18:06:59.087304    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ad18d44_3fa9_4b61_a206_c0195cabc7c7.slice/crio-0746c60579f15bd908fd87d5a8658a6e839da3d5880a398221f34a1719bd5fe0 WatchSource:0}: Error finding container 0746c60579f15bd908fd87d5a8658a6e839da3d5880a398221f34a1719bd5fe0: Status 404 returned error can't find the container with id 0746c60579f15bd908fd87d5a8658a6e839da3d5880a398221f34a1719bd5fe0
Aug 04 18:07:00.035889 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:07:00.035860    2583 generic.go:356] "Generic (PLEG): container finished" podID="c6e0e155-dddf-427c-a55c-bee6b6b0fbfd" containerID="c987663fe6d1e193e6224e06fbeab9f0d4c6336817d7609428245b1b5e50d156" exitCode=0
Aug 04 18:07:00.036234 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:07:00.035944    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c29trg" event={"ID":"c6e0e155-dddf-427c-a55c-bee6b6b0fbfd","Type":"ContainerDied","Data":"c987663fe6d1e193e6224e06fbeab9f0d4c6336817d7609428245b1b5e50d156"}
Aug 04 18:07:00.037434 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:07:00.037396    2583 generic.go:356] "Generic (PLEG): container finished" podID="5ad18d44-3fa9-4b61-a206-c0195cabc7c7" containerID="4c21dfc9eee07065c923a6f3878393fa69ff5a1dfd000745a883cf2bec21e051" exitCode=0
Aug 04 18:07:00.037532 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:07:00.037494    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0p2s68" event={"ID":"5ad18d44-3fa9-4b61-a206-c0195cabc7c7","Type":"ContainerDied","Data":"4c21dfc9eee07065c923a6f3878393fa69ff5a1dfd000745a883cf2bec21e051"}
Aug 04 18:07:00.037598 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:07:00.037543    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0p2s68" event={"ID":"5ad18d44-3fa9-4b61-a206-c0195cabc7c7","Type":"ContainerStarted","Data":"0746c60579f15bd908fd87d5a8658a6e839da3d5880a398221f34a1719bd5fe0"}
Aug 04 18:07:00.039115 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:07:00.039095    2583 generic.go:356] "Generic (PLEG): container finished" podID="bd9ea619-2bb1-42d9-a67b-3331ad969d81" containerID="00a2b36b876cf07d366fe28e3953cd7f6bc412f8cff82b9d628ee8c4b426f7b7" exitCode=0
Aug 04 18:07:00.039275 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:07:00.039194    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb6llcz" event={"ID":"bd9ea619-2bb1-42d9-a67b-3331ad969d81","Type":"ContainerDied","Data":"00a2b36b876cf07d366fe28e3953cd7f6bc412f8cff82b9d628ee8c4b426f7b7"}
Aug 04 18:07:00.041410 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:07:00.041389    2583 generic.go:356] "Generic (PLEG): container finished" podID="d44b74d5-56cf-423a-8bf6-8c325024a661" containerID="9f006fcbe92529cc8b2584101af95b204f877ade4d2b7f4b6b571269c82849dc" exitCode=0
Aug 04 18:07:00.041535 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:07:00.041433    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870539ksjk" event={"ID":"d44b74d5-56cf-423a-8bf6-8c325024a661","Type":"ContainerDied","Data":"9f006fcbe92529cc8b2584101af95b204f877ade4d2b7f4b6b571269c82849dc"}
Aug 04 18:07:01.055086 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:07:01.055051    2583 generic.go:356] "Generic (PLEG): container finished" podID="c6e0e155-dddf-427c-a55c-bee6b6b0fbfd" containerID="868bc86b4ed108f797696410dd03a218819a96f57e794f14f0ccce4da3678a92" exitCode=0
Aug 04 18:07:01.055551 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:07:01.055131    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c29trg" event={"ID":"c6e0e155-dddf-427c-a55c-bee6b6b0fbfd","Type":"ContainerDied","Data":"868bc86b4ed108f797696410dd03a218819a96f57e794f14f0ccce4da3678a92"}
Aug 04 18:07:01.056821 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:07:01.056797    2583 generic.go:356] "Generic (PLEG): container finished" podID="5ad18d44-3fa9-4b61-a206-c0195cabc7c7" containerID="1f0959b5aa53c76db3ff8539ba6ec17f488ea293e55fb72203a53cb66d3708f6" exitCode=0
Aug 04 18:07:01.056943 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:07:01.056878    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0p2s68" event={"ID":"5ad18d44-3fa9-4b61-a206-c0195cabc7c7","Type":"ContainerDied","Data":"1f0959b5aa53c76db3ff8539ba6ec17f488ea293e55fb72203a53cb66d3708f6"}
Aug 04 18:07:01.058930 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:07:01.058911    2583 generic.go:356] "Generic (PLEG): container finished" podID="d44b74d5-56cf-423a-8bf6-8c325024a661" containerID="5bed519bc82d5577af82ed0b081b01853594c712442f52c7477795a70fdcb3ad" exitCode=0
Aug 04 18:07:01.059017 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:07:01.058940    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870539ksjk" event={"ID":"d44b74d5-56cf-423a-8bf6-8c325024a661","Type":"ContainerDied","Data":"5bed519bc82d5577af82ed0b081b01853594c712442f52c7477795a70fdcb3ad"}
Aug 04 18:07:02.066402 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:07:02.066321    2583 generic.go:356] "Generic (PLEG): container finished" podID="5ad18d44-3fa9-4b61-a206-c0195cabc7c7" containerID="2a8634eb450c964554d64281588290b5eedbf75a2019bc48c269f99c042422aa" exitCode=0
Aug 04 18:07:02.066805 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:07:02.066408    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0p2s68" event={"ID":"5ad18d44-3fa9-4b61-a206-c0195cabc7c7","Type":"ContainerDied","Data":"2a8634eb450c964554d64281588290b5eedbf75a2019bc48c269f99c042422aa"}
Aug 04 18:07:02.068290 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:07:02.068265    2583 generic.go:356] "Generic (PLEG): container finished" podID="bd9ea619-2bb1-42d9-a67b-3331ad969d81" containerID="7205b40445bd5821ae4c5b5e526b8ec8ece235c560f025cc954b855935bb2c17" exitCode=0
Aug 04 18:07:02.068405 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:07:02.068338    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb6llcz" event={"ID":"bd9ea619-2bb1-42d9-a67b-3331ad969d81","Type":"ContainerDied","Data":"7205b40445bd5821ae4c5b5e526b8ec8ece235c560f025cc954b855935bb2c17"}
Aug 04 18:07:02.278626 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:07:02.278600    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870539ksjk"
Aug 04 18:07:02.281154 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:07:02.281133    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c29trg"
Aug 04 18:07:02.345292 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:07:02.345208    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kckxx\" (UniqueName: \"kubernetes.io/projected/d44b74d5-56cf-423a-8bf6-8c325024a661-kube-api-access-kckxx\") pod \"d44b74d5-56cf-423a-8bf6-8c325024a661\" (UID: \"d44b74d5-56cf-423a-8bf6-8c325024a661\") "
Aug 04 18:07:02.345292 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:07:02.345258    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c6e0e155-dddf-427c-a55c-bee6b6b0fbfd-util\") pod \"c6e0e155-dddf-427c-a55c-bee6b6b0fbfd\" (UID: \"c6e0e155-dddf-427c-a55c-bee6b6b0fbfd\") "
Aug 04 18:07:02.345292 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:07:02.345279    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c6e0e155-dddf-427c-a55c-bee6b6b0fbfd-bundle\") pod \"c6e0e155-dddf-427c-a55c-bee6b6b0fbfd\" (UID: \"c6e0e155-dddf-427c-a55c-bee6b6b0fbfd\") "
Aug 04 18:07:02.345591 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:07:02.345294    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d44b74d5-56cf-423a-8bf6-8c325024a661-util\") pod \"d44b74d5-56cf-423a-8bf6-8c325024a661\" (UID: \"d44b74d5-56cf-423a-8bf6-8c325024a661\") "
Aug 04 18:07:02.345591 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:07:02.345322    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ql7f6\" (UniqueName: \"kubernetes.io/projected/c6e0e155-dddf-427c-a55c-bee6b6b0fbfd-kube-api-access-ql7f6\") pod \"c6e0e155-dddf-427c-a55c-bee6b6b0fbfd\" (UID: \"c6e0e155-dddf-427c-a55c-bee6b6b0fbfd\") "
Aug 04 18:07:02.345591 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:07:02.345372    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d44b74d5-56cf-423a-8bf6-8c325024a661-bundle\") pod \"d44b74d5-56cf-423a-8bf6-8c325024a661\" (UID: \"d44b74d5-56cf-423a-8bf6-8c325024a661\") "
Aug 04 18:07:02.345959 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:07:02.345932    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6e0e155-dddf-427c-a55c-bee6b6b0fbfd-bundle" (OuterVolumeSpecName: "bundle") pod "c6e0e155-dddf-427c-a55c-bee6b6b0fbfd" (UID: "c6e0e155-dddf-427c-a55c-bee6b6b0fbfd"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:07:02.346167 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:07:02.346143    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d44b74d5-56cf-423a-8bf6-8c325024a661-bundle" (OuterVolumeSpecName: "bundle") pod "d44b74d5-56cf-423a-8bf6-8c325024a661" (UID: "d44b74d5-56cf-423a-8bf6-8c325024a661"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:07:02.347873 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:07:02.347852    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6e0e155-dddf-427c-a55c-bee6b6b0fbfd-kube-api-access-ql7f6" (OuterVolumeSpecName: "kube-api-access-ql7f6") pod "c6e0e155-dddf-427c-a55c-bee6b6b0fbfd" (UID: "c6e0e155-dddf-427c-a55c-bee6b6b0fbfd"). InnerVolumeSpecName "kube-api-access-ql7f6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:07:02.347972 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:07:02.347868    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d44b74d5-56cf-423a-8bf6-8c325024a661-kube-api-access-kckxx" (OuterVolumeSpecName: "kube-api-access-kckxx") pod "d44b74d5-56cf-423a-8bf6-8c325024a661" (UID: "d44b74d5-56cf-423a-8bf6-8c325024a661"). InnerVolumeSpecName "kube-api-access-kckxx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:07:02.351323 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:07:02.351301    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6e0e155-dddf-427c-a55c-bee6b6b0fbfd-util" (OuterVolumeSpecName: "util") pod "c6e0e155-dddf-427c-a55c-bee6b6b0fbfd" (UID: "c6e0e155-dddf-427c-a55c-bee6b6b0fbfd"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:07:02.351774 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:07:02.351750    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d44b74d5-56cf-423a-8bf6-8c325024a661-util" (OuterVolumeSpecName: "util") pod "d44b74d5-56cf-423a-8bf6-8c325024a661" (UID: "d44b74d5-56cf-423a-8bf6-8c325024a661"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:07:02.446023 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:07:02.445995    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kckxx\" (UniqueName: \"kubernetes.io/projected/d44b74d5-56cf-423a-8bf6-8c325024a661-kube-api-access-kckxx\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:07:02.446023 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:07:02.446020    2583 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c6e0e155-dddf-427c-a55c-bee6b6b0fbfd-util\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:07:02.446023 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:07:02.446030    2583 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c6e0e155-dddf-427c-a55c-bee6b6b0fbfd-bundle\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:07:02.446338 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:07:02.446039    2583 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d44b74d5-56cf-423a-8bf6-8c325024a661-util\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:07:02.446338 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:07:02.446046    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ql7f6\" (UniqueName: \"kubernetes.io/projected/c6e0e155-dddf-427c-a55c-bee6b6b0fbfd-kube-api-access-ql7f6\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:07:02.446338 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:07:02.446054    2583 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d44b74d5-56cf-423a-8bf6-8c325024a661-bundle\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:07:03.074707 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:07:03.074626    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c29trg"
Aug 04 18:07:03.075094 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:07:03.074626    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183c29trg" event={"ID":"c6e0e155-dddf-427c-a55c-bee6b6b0fbfd","Type":"ContainerDied","Data":"9ae405446806c9d0f216c3ae6646589a4a190aaacdf33b9d80ca10651d4409fe"}
Aug 04 18:07:03.075094 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:07:03.074754    2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9ae405446806c9d0f216c3ae6646589a4a190aaacdf33b9d80ca10651d4409fe"
Aug 04 18:07:03.076758 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:07:03.076729    2583 generic.go:356] "Generic (PLEG): container finished" podID="bd9ea619-2bb1-42d9-a67b-3331ad969d81" containerID="a91157b9fb2099fce0f170edd15bdb34aa68dc106ef67333e915fd7152c17d81" exitCode=0
Aug 04 18:07:03.076933 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:07:03.076842    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb6llcz" event={"ID":"bd9ea619-2bb1-42d9-a67b-3331ad969d81","Type":"ContainerDied","Data":"a91157b9fb2099fce0f170edd15bdb34aa68dc106ef67333e915fd7152c17d81"}
Aug 04 18:07:03.078634 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:07:03.078611    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870539ksjk"
Aug 04 18:07:03.078707 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:07:03.078634    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d870539ksjk" event={"ID":"d44b74d5-56cf-423a-8bf6-8c325024a661","Type":"ContainerDied","Data":"259f50c458c03606e126bfa9b3e008d0e43a0fe55e6fa2bf376e4445482c9f45"}
Aug 04 18:07:03.078707 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:07:03.078669    2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="259f50c458c03606e126bfa9b3e008d0e43a0fe55e6fa2bf376e4445482c9f45"
Aug 04 18:07:03.211183 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:07:03.211157    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0p2s68"
Aug 04 18:07:03.354465 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:07:03.354369    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zmjt7\" (UniqueName: \"kubernetes.io/projected/5ad18d44-3fa9-4b61-a206-c0195cabc7c7-kube-api-access-zmjt7\") pod \"5ad18d44-3fa9-4b61-a206-c0195cabc7c7\" (UID: \"5ad18d44-3fa9-4b61-a206-c0195cabc7c7\") "
Aug 04 18:07:03.354465 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:07:03.354427    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5ad18d44-3fa9-4b61-a206-c0195cabc7c7-bundle\") pod \"5ad18d44-3fa9-4b61-a206-c0195cabc7c7\" (UID: \"5ad18d44-3fa9-4b61-a206-c0195cabc7c7\") "
Aug 04 18:07:03.354685 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:07:03.354521    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5ad18d44-3fa9-4b61-a206-c0195cabc7c7-util\") pod \"5ad18d44-3fa9-4b61-a206-c0195cabc7c7\" (UID: \"5ad18d44-3fa9-4b61-a206-c0195cabc7c7\") "
Aug 04 18:07:03.354987 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:07:03.354955    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ad18d44-3fa9-4b61-a206-c0195cabc7c7-bundle" (OuterVolumeSpecName: "bundle") pod "5ad18d44-3fa9-4b61-a206-c0195cabc7c7" (UID: "5ad18d44-3fa9-4b61-a206-c0195cabc7c7"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:07:03.356501 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:07:03.356450    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ad18d44-3fa9-4b61-a206-c0195cabc7c7-kube-api-access-zmjt7" (OuterVolumeSpecName: "kube-api-access-zmjt7") pod "5ad18d44-3fa9-4b61-a206-c0195cabc7c7" (UID: "5ad18d44-3fa9-4b61-a206-c0195cabc7c7"). InnerVolumeSpecName "kube-api-access-zmjt7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:07:03.359376 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:07:03.359345    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ad18d44-3fa9-4b61-a206-c0195cabc7c7-util" (OuterVolumeSpecName: "util") pod "5ad18d44-3fa9-4b61-a206-c0195cabc7c7" (UID: "5ad18d44-3fa9-4b61-a206-c0195cabc7c7"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:07:03.455142 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:07:03.455112    2583 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5ad18d44-3fa9-4b61-a206-c0195cabc7c7-util\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:07:03.455142 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:07:03.455137    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zmjt7\" (UniqueName: \"kubernetes.io/projected/5ad18d44-3fa9-4b61-a206-c0195cabc7c7-kube-api-access-zmjt7\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:07:03.455142 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:07:03.455146    2583 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5ad18d44-3fa9-4b61-a206-c0195cabc7c7-bundle\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:07:04.092360 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:07:04.092323    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0p2s68" event={"ID":"5ad18d44-3fa9-4b61-a206-c0195cabc7c7","Type":"ContainerDied","Data":"0746c60579f15bd908fd87d5a8658a6e839da3d5880a398221f34a1719bd5fe0"}
Aug 04 18:07:04.092360 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:07:04.092352    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0p2s68"
Aug 04 18:07:04.092855 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:07:04.092404    2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0746c60579f15bd908fd87d5a8658a6e839da3d5880a398221f34a1719bd5fe0"
Aug 04 18:07:04.228116 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:07:04.228091    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb6llcz"
Aug 04 18:07:04.362339 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:07:04.362247    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qc8w2\" (UniqueName: \"kubernetes.io/projected/bd9ea619-2bb1-42d9-a67b-3331ad969d81-kube-api-access-qc8w2\") pod \"bd9ea619-2bb1-42d9-a67b-3331ad969d81\" (UID: \"bd9ea619-2bb1-42d9-a67b-3331ad969d81\") "
Aug 04 18:07:04.362536 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:07:04.362348    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bd9ea619-2bb1-42d9-a67b-3331ad969d81-util\") pod \"bd9ea619-2bb1-42d9-a67b-3331ad969d81\" (UID: \"bd9ea619-2bb1-42d9-a67b-3331ad969d81\") "
Aug 04 18:07:04.362536 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:07:04.362374    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bd9ea619-2bb1-42d9-a67b-3331ad969d81-bundle\") pod \"bd9ea619-2bb1-42d9-a67b-3331ad969d81\" (UID: \"bd9ea619-2bb1-42d9-a67b-3331ad969d81\") "
Aug 04 18:07:04.362888 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:07:04.362862    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd9ea619-2bb1-42d9-a67b-3331ad969d81-bundle" (OuterVolumeSpecName: "bundle") pod "bd9ea619-2bb1-42d9-a67b-3331ad969d81" (UID: "bd9ea619-2bb1-42d9-a67b-3331ad969d81"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:07:04.364331 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:07:04.364312    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd9ea619-2bb1-42d9-a67b-3331ad969d81-kube-api-access-qc8w2" (OuterVolumeSpecName: "kube-api-access-qc8w2") pod "bd9ea619-2bb1-42d9-a67b-3331ad969d81" (UID: "bd9ea619-2bb1-42d9-a67b-3331ad969d81"). InnerVolumeSpecName "kube-api-access-qc8w2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:07:04.367053 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:07:04.367027    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd9ea619-2bb1-42d9-a67b-3331ad969d81-util" (OuterVolumeSpecName: "util") pod "bd9ea619-2bb1-42d9-a67b-3331ad969d81" (UID: "bd9ea619-2bb1-42d9-a67b-3331ad969d81"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:07:04.463350 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:07:04.463319    2583 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bd9ea619-2bb1-42d9-a67b-3331ad969d81-util\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:07:04.463350 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:07:04.463345    2583 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bd9ea619-2bb1-42d9-a67b-3331ad969d81-bundle\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:07:04.463350 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:07:04.463354    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qc8w2\" (UniqueName: \"kubernetes.io/projected/bd9ea619-2bb1-42d9-a67b-3331ad969d81-kube-api-access-qc8w2\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:07:05.098313 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:07:05.098285    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb6llcz"
Aug 04 18:07:05.098313 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:07:05.098298    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb6llcz" event={"ID":"bd9ea619-2bb1-42d9-a67b-3331ad969d81","Type":"ContainerDied","Data":"0bf4ba6ac65ab185d59058fed70408387fe97ddeabb1005d5369ce27117e8a89"}
Aug 04 18:07:05.098764 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:07:05.098345    2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0bf4ba6ac65ab185d59058fed70408387fe97ddeabb1005d5369ce27117e8a89"
Aug 04 18:07:11.810775 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:07:11.810697    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-gr7sm"]
Aug 04 18:07:11.811120 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:07:11.811004    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd9ea619-2bb1-42d9-a67b-3331ad969d81" containerName="pull"
Aug 04 18:07:11.811120 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:07:11.811039    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd9ea619-2bb1-42d9-a67b-3331ad969d81" containerName="pull"
Aug 04 18:07:11.811120 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:07:11.811052    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ad18d44-3fa9-4b61-a206-c0195cabc7c7" containerName="pull"
Aug 04 18:07:11.811120 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:07:11.811057    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ad18d44-3fa9-4b61-a206-c0195cabc7c7" containerName="pull"
Aug 04 18:07:11.811120 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:07:11.811065    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6e0e155-dddf-427c-a55c-bee6b6b0fbfd" containerName="util"
Aug 04 18:07:11.811120 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:07:11.811070    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6e0e155-dddf-427c-a55c-bee6b6b0fbfd" containerName="util"
Aug 04 18:07:11.811120 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:07:11.811076    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ad18d44-3fa9-4b61-a206-c0195cabc7c7" containerName="util"
Aug 04 18:07:11.811120 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:07:11.811082    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ad18d44-3fa9-4b61-a206-c0195cabc7c7" containerName="util"
Aug 04 18:07:11.811120 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:07:11.811091    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd9ea619-2bb1-42d9-a67b-3331ad969d81" containerName="extract"
Aug 04 18:07:11.811120 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:07:11.811098    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd9ea619-2bb1-42d9-a67b-3331ad969d81" containerName="extract"
Aug 04 18:07:11.811120 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:07:11.811107    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d44b74d5-56cf-423a-8bf6-8c325024a661" containerName="util"
Aug 04 18:07:11.811120 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:07:11.811112    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="d44b74d5-56cf-423a-8bf6-8c325024a661" containerName="util"
Aug 04 18:07:11.811120 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:07:11.811118    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd9ea619-2bb1-42d9-a67b-3331ad969d81" containerName="util"
Aug 04 18:07:11.811120 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:07:11.811123    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd9ea619-2bb1-42d9-a67b-3331ad969d81" containerName="util"
Aug 04 18:07:11.811120 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:07:11.811129    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6e0e155-dddf-427c-a55c-bee6b6b0fbfd" containerName="extract"
Aug 04 18:07:11.811599 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:07:11.811134    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6e0e155-dddf-427c-a55c-bee6b6b0fbfd" containerName="extract"
Aug 04 18:07:11.811599 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:07:11.811141    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d44b74d5-56cf-423a-8bf6-8c325024a661" containerName="pull"
Aug 04 18:07:11.811599 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:07:11.811146    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="d44b74d5-56cf-423a-8bf6-8c325024a661" containerName="pull"
Aug 04 18:07:11.811599 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:07:11.811153    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ad18d44-3fa9-4b61-a206-c0195cabc7c7" containerName="extract"
Aug 04 18:07:11.811599 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:07:11.811158    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ad18d44-3fa9-4b61-a206-c0195cabc7c7" containerName="extract"
Aug 04 18:07:11.811599 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:07:11.811166    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6e0e155-dddf-427c-a55c-bee6b6b0fbfd" containerName="pull"
Aug 04 18:07:11.811599 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:07:11.811171    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6e0e155-dddf-427c-a55c-bee6b6b0fbfd" containerName="pull"
Aug 04 18:07:11.811599 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:07:11.811177    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d44b74d5-56cf-423a-8bf6-8c325024a661" containerName="extract"
Aug 04 18:07:11.811599 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:07:11.811181    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="d44b74d5-56cf-423a-8bf6-8c325024a661" containerName="extract"
Aug 04 18:07:11.811599 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:07:11.811233    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6e0e155-dddf-427c-a55c-bee6b6b0fbfd" containerName="extract"
Aug 04 18:07:11.811599 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:07:11.811240    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="d44b74d5-56cf-423a-8bf6-8c325024a661" containerName="extract"
Aug 04 18:07:11.811599 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:07:11.811247    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ad18d44-3fa9-4b61-a206-c0195cabc7c7" containerName="extract"
Aug 04 18:07:11.811599 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:07:11.811253    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd9ea619-2bb1-42d9-a67b-3331ad969d81" containerName="extract"
Aug 04 18:07:11.814048 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:07:11.814031    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-gr7sm"
Aug 04 18:07:11.818190 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:07:11.818169    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-ctpxf\""
Aug 04 18:07:11.818301 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:07:11.818169    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Aug 04 18:07:11.818301 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:07:11.818169    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Aug 04 18:07:11.831831 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:07:11.831804    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-gr7sm"]
Aug 04 18:07:11.925564 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:07:11.925528    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/69c87acd-2bcc-454d-93bc-2ddb9f32d3ea-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-66866f5fcc-gr7sm\" (UID: \"69c87acd-2bcc-454d-93bc-2ddb9f32d3ea\") " pod="kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-gr7sm"
Aug 04 18:07:11.925564 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:07:11.925561    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zf4dz\" (UniqueName: \"kubernetes.io/projected/69c87acd-2bcc-454d-93bc-2ddb9f32d3ea-kube-api-access-zf4dz\") pod \"kuadrant-operator-controller-manager-66866f5fcc-gr7sm\" (UID: \"69c87acd-2bcc-454d-93bc-2ddb9f32d3ea\") " pod="kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-gr7sm"
Aug 04 18:07:12.026213 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:07:12.026172    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/69c87acd-2bcc-454d-93bc-2ddb9f32d3ea-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-66866f5fcc-gr7sm\" (UID: \"69c87acd-2bcc-454d-93bc-2ddb9f32d3ea\") " pod="kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-gr7sm"
Aug 04 18:07:12.026213 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:07:12.026211    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zf4dz\" (UniqueName: \"kubernetes.io/projected/69c87acd-2bcc-454d-93bc-2ddb9f32d3ea-kube-api-access-zf4dz\") pod \"kuadrant-operator-controller-manager-66866f5fcc-gr7sm\" (UID: \"69c87acd-2bcc-454d-93bc-2ddb9f32d3ea\") " pod="kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-gr7sm"
Aug 04 18:07:12.026657 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:07:12.026634    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/69c87acd-2bcc-454d-93bc-2ddb9f32d3ea-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-66866f5fcc-gr7sm\" (UID: \"69c87acd-2bcc-454d-93bc-2ddb9f32d3ea\") " pod="kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-gr7sm"
Aug 04 18:07:12.035306 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:07:12.035278    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zf4dz\" (UniqueName: \"kubernetes.io/projected/69c87acd-2bcc-454d-93bc-2ddb9f32d3ea-kube-api-access-zf4dz\") pod \"kuadrant-operator-controller-manager-66866f5fcc-gr7sm\" (UID: \"69c87acd-2bcc-454d-93bc-2ddb9f32d3ea\") " pod="kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-gr7sm"
Aug 04 18:07:12.124183 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:07:12.124096    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-gr7sm"
Aug 04 18:07:12.263743 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:07:12.263717    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-gr7sm"]
Aug 04 18:07:12.265822 ip-10-0-135-124 kubenswrapper[2583]: W0804 18:07:12.265793    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69c87acd_2bcc_454d_93bc_2ddb9f32d3ea.slice/crio-de7e3f6ae980725b41e802bfdf1bf8d8f30c2d74e516d01dfcab9c31101a1569 WatchSource:0}: Error finding container de7e3f6ae980725b41e802bfdf1bf8d8f30c2d74e516d01dfcab9c31101a1569: Status 404 returned error can't find the container with id de7e3f6ae980725b41e802bfdf1bf8d8f30c2d74e516d01dfcab9c31101a1569
Aug 04 18:07:13.140286 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:07:13.140245    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-gr7sm" event={"ID":"69c87acd-2bcc-454d-93bc-2ddb9f32d3ea","Type":"ContainerStarted","Data":"de7e3f6ae980725b41e802bfdf1bf8d8f30c2d74e516d01dfcab9c31101a1569"}
Aug 04 18:07:13.812866 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:07:13.812639    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-ccbdf686-h9lf5"]
Aug 04 18:07:13.816142 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:07:13.816118    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-ccbdf686-h9lf5"
Aug 04 18:07:13.819167 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:07:13.819139    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Aug 04 18:07:13.819262 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:07:13.819241    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-qx67n\""
Aug 04 18:07:13.828208 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:07:13.828184    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-ccbdf686-h9lf5"]
Aug 04 18:07:13.944648 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:07:13.944612    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfrv7\" (UniqueName: \"kubernetes.io/projected/565c38b3-e6f6-45f9-9c06-c3e84f642f58-kube-api-access-bfrv7\") pod \"dns-operator-controller-manager-ccbdf686-h9lf5\" (UID: \"565c38b3-e6f6-45f9-9c06-c3e84f642f58\") " pod="kuadrant-system/dns-operator-controller-manager-ccbdf686-h9lf5"
Aug 04 18:07:14.045630 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:07:14.045573    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfrv7\" (UniqueName: \"kubernetes.io/projected/565c38b3-e6f6-45f9-9c06-c3e84f642f58-kube-api-access-bfrv7\") pod \"dns-operator-controller-manager-ccbdf686-h9lf5\" (UID: \"565c38b3-e6f6-45f9-9c06-c3e84f642f58\") " pod="kuadrant-system/dns-operator-controller-manager-ccbdf686-h9lf5"
Aug 04 18:07:14.055422 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:07:14.055357    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfrv7\" (UniqueName: \"kubernetes.io/projected/565c38b3-e6f6-45f9-9c06-c3e84f642f58-kube-api-access-bfrv7\") pod \"dns-operator-controller-manager-ccbdf686-h9lf5\" (UID: \"565c38b3-e6f6-45f9-9c06-c3e84f642f58\") " pod="kuadrant-system/dns-operator-controller-manager-ccbdf686-h9lf5"
Aug 04 18:07:14.130743 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:07:14.130656    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-ccbdf686-h9lf5"
Aug 04 18:07:14.827573 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:07:14.827541    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-ccbdf686-h9lf5"]
Aug 04 18:07:14.828124 ip-10-0-135-124 kubenswrapper[2583]: W0804 18:07:14.828097    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod565c38b3_e6f6_45f9_9c06_c3e84f642f58.slice/crio-350d6e0c291947306aa2482b652dd431f0281bd576ec5925a1f7aeed93248af3 WatchSource:0}: Error finding container 350d6e0c291947306aa2482b652dd431f0281bd576ec5925a1f7aeed93248af3: Status 404 returned error can't find the container with id 350d6e0c291947306aa2482b652dd431f0281bd576ec5925a1f7aeed93248af3
Aug 04 18:07:15.152929 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:07:15.152891    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-ccbdf686-h9lf5" event={"ID":"565c38b3-e6f6-45f9-9c06-c3e84f642f58","Type":"ContainerStarted","Data":"350d6e0c291947306aa2482b652dd431f0281bd576ec5925a1f7aeed93248af3"}
Aug 04 18:07:16.432886 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:07:16.432849    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-56fb6f6856-m89z5"]
Aug 04 18:07:16.436958 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:07:16.436935    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-56fb6f6856-m89z5"
Aug 04 18:07:16.440312 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:07:16.440285    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-bsssh\""
Aug 04 18:07:16.448984 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:07:16.448953    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-56fb6f6856-m89z5"]
Aug 04 18:07:16.569224 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:07:16.569173    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76vwk\" (UniqueName: \"kubernetes.io/projected/52e21e58-78e1-499b-a335-8e4991d308b0-kube-api-access-76vwk\") pod \"authorino-operator-56fb6f6856-m89z5\" (UID: \"52e21e58-78e1-499b-a335-8e4991d308b0\") " pod="kuadrant-system/authorino-operator-56fb6f6856-m89z5"
Aug 04 18:07:16.670386 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:07:16.670349    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76vwk\" (UniqueName: \"kubernetes.io/projected/52e21e58-78e1-499b-a335-8e4991d308b0-kube-api-access-76vwk\") pod \"authorino-operator-56fb6f6856-m89z5\" (UID: \"52e21e58-78e1-499b-a335-8e4991d308b0\") " pod="kuadrant-system/authorino-operator-56fb6f6856-m89z5"
Aug 04 18:07:16.681143 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:07:16.681117    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-76vwk\" (UniqueName: \"kubernetes.io/projected/52e21e58-78e1-499b-a335-8e4991d308b0-kube-api-access-76vwk\") pod \"authorino-operator-56fb6f6856-m89z5\" (UID: \"52e21e58-78e1-499b-a335-8e4991d308b0\") " pod="kuadrant-system/authorino-operator-56fb6f6856-m89z5"
Aug 04 18:07:16.750440 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:07:16.750354    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-56fb6f6856-m89z5"
Aug 04 18:07:16.915157 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:07:16.915128    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-56fb6f6856-m89z5"]
Aug 04 18:07:16.916865 ip-10-0-135-124 kubenswrapper[2583]: W0804 18:07:16.916839    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52e21e58_78e1_499b_a335_8e4991d308b0.slice/crio-f236c82341225653319f5f7506925d6718c4c9f540044a23254f85127d6ec967 WatchSource:0}: Error finding container f236c82341225653319f5f7506925d6718c4c9f540044a23254f85127d6ec967: Status 404 returned error can't find the container with id f236c82341225653319f5f7506925d6718c4c9f540044a23254f85127d6ec967
Aug 04 18:07:17.165911 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:07:17.165821    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-gr7sm" event={"ID":"69c87acd-2bcc-454d-93bc-2ddb9f32d3ea","Type":"ContainerStarted","Data":"31a0055599ca637dd2c327b911c9cb92faa45766d61c2a3e9112cd9f14b56e74"}
Aug 04 18:07:17.166155 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:07:17.165957    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-gr7sm"
Aug 04 18:07:17.167267 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:07:17.167241    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-56fb6f6856-m89z5" event={"ID":"52e21e58-78e1-499b-a335-8e4991d308b0","Type":"ContainerStarted","Data":"f236c82341225653319f5f7506925d6718c4c9f540044a23254f85127d6ec967"}
Aug 04 18:07:17.197575 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:07:17.197441    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-gr7sm" podStartSLOduration=1.712852286 podStartE2EDuration="6.197427097s" totalImagesPullingTime="4.484574811s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:07:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:07:17.197290556 +0000 UTC m=+450.778509137" watchObservedRunningTime="2026-08-04 18:07:17.197427097 +0000 UTC m=+450.778645685"
Aug 04 18:07:19.179795 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:07:19.179701    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-56fb6f6856-m89z5" event={"ID":"52e21e58-78e1-499b-a335-8e4991d308b0","Type":"ContainerStarted","Data":"648b2f9d5b782c5dc6e4eb353d4005abae169cbfab431684a558761b891c06da"}
Aug 04 18:07:19.179795 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:07:19.179778    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-56fb6f6856-m89z5"
Aug 04 18:07:19.200710 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:07:19.200652    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-56fb6f6856-m89z5" podStartSLOduration=1.262284009 podStartE2EDuration="3.200635794s" totalImagesPullingTime="1.938351785s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:07:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:07:19.199277941 +0000 UTC m=+452.780496518" watchObservedRunningTime="2026-08-04 18:07:19.200635794 +0000 UTC m=+452.781854366"
Aug 04 18:07:19.503288 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:07:19.503249    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-598f86468b-tjxk2"]
Aug 04 18:07:19.506725 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:07:19.506701    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-tjxk2"
Aug 04 18:07:19.509618 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:07:19.509594    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/e88b512e-53a6-4cbd-9349-60d49c02f10e-nginx-conf\") pod \"kuadrant-console-plugin-598f86468b-tjxk2\" (UID: \"e88b512e-53a6-4cbd-9349-60d49c02f10e\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-tjxk2"
Aug 04 18:07:19.509714 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:07:19.509672    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hj2f\" (UniqueName: \"kubernetes.io/projected/e88b512e-53a6-4cbd-9349-60d49c02f10e-kube-api-access-9hj2f\") pod \"kuadrant-console-plugin-598f86468b-tjxk2\" (UID: \"e88b512e-53a6-4cbd-9349-60d49c02f10e\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-tjxk2"
Aug 04 18:07:19.509772 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:07:19.509720    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/e88b512e-53a6-4cbd-9349-60d49c02f10e-plugin-serving-cert\") pod \"kuadrant-console-plugin-598f86468b-tjxk2\" (UID: \"e88b512e-53a6-4cbd-9349-60d49c02f10e\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-tjxk2"
Aug 04 18:07:19.511494 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:07:19.511457    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Aug 04 18:07:19.513625 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:07:19.513417    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Aug 04 18:07:19.530594 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:07:19.530563    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-g7sl8\""
Aug 04 18:07:19.532982 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:07:19.532952    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-598f86468b-tjxk2"]
Aug 04 18:07:19.610974 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:07:19.610941    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/e88b512e-53a6-4cbd-9349-60d49c02f10e-nginx-conf\") pod \"kuadrant-console-plugin-598f86468b-tjxk2\" (UID: \"e88b512e-53a6-4cbd-9349-60d49c02f10e\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-tjxk2"
Aug 04 18:07:19.611160 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:07:19.611008    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hj2f\" (UniqueName: \"kubernetes.io/projected/e88b512e-53a6-4cbd-9349-60d49c02f10e-kube-api-access-9hj2f\") pod \"kuadrant-console-plugin-598f86468b-tjxk2\" (UID: \"e88b512e-53a6-4cbd-9349-60d49c02f10e\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-tjxk2"
Aug 04 18:07:19.611160 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:07:19.611052    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/e88b512e-53a6-4cbd-9349-60d49c02f10e-plugin-serving-cert\") pod \"kuadrant-console-plugin-598f86468b-tjxk2\" (UID: \"e88b512e-53a6-4cbd-9349-60d49c02f10e\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-tjxk2"
Aug 04 18:07:19.611779 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:07:19.611749    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/e88b512e-53a6-4cbd-9349-60d49c02f10e-nginx-conf\") pod \"kuadrant-console-plugin-598f86468b-tjxk2\" (UID: \"e88b512e-53a6-4cbd-9349-60d49c02f10e\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-tjxk2"
Aug 04 18:07:19.613698 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:07:19.613676    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/e88b512e-53a6-4cbd-9349-60d49c02f10e-plugin-serving-cert\") pod \"kuadrant-console-plugin-598f86468b-tjxk2\" (UID: \"e88b512e-53a6-4cbd-9349-60d49c02f10e\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-tjxk2"
Aug 04 18:07:19.619849 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:07:19.619821    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hj2f\" (UniqueName: \"kubernetes.io/projected/e88b512e-53a6-4cbd-9349-60d49c02f10e-kube-api-access-9hj2f\") pod \"kuadrant-console-plugin-598f86468b-tjxk2\" (UID: \"e88b512e-53a6-4cbd-9349-60d49c02f10e\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-tjxk2"
Aug 04 18:07:19.822347 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:07:19.822255    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-tjxk2"
Aug 04 18:07:20.042977 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:07:20.042950    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-598f86468b-tjxk2"]
Aug 04 18:07:20.045854 ip-10-0-135-124 kubenswrapper[2583]: W0804 18:07:20.045818    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode88b512e_53a6_4cbd_9349_60d49c02f10e.slice/crio-4b84f29897b91ba95bb09c65deac3a3819f4bf4df548587832697f7fec1ddcbb WatchSource:0}: Error finding container 4b84f29897b91ba95bb09c65deac3a3819f4bf4df548587832697f7fec1ddcbb: Status 404 returned error can't find the container with id 4b84f29897b91ba95bb09c65deac3a3819f4bf4df548587832697f7fec1ddcbb
Aug 04 18:07:20.186796 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:07:20.186762    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-tjxk2" event={"ID":"e88b512e-53a6-4cbd-9349-60d49c02f10e","Type":"ContainerStarted","Data":"4b84f29897b91ba95bb09c65deac3a3819f4bf4df548587832697f7fec1ddcbb"}
Aug 04 18:07:20.188427 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:07:20.188403    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-ccbdf686-h9lf5" event={"ID":"565c38b3-e6f6-45f9-9c06-c3e84f642f58","Type":"ContainerStarted","Data":"727c7e54eaadb28ebca37d58132314f0e86796ceea41bf89da71d162244b1023"}
Aug 04 18:07:20.217721 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:07:20.217677    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-ccbdf686-h9lf5" podStartSLOduration=2.066399106 podStartE2EDuration="7.217662448s" totalImagesPullingTime="5.151263342s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:07:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:07:20.21536266 +0000 UTC m=+453.796581237" watchObservedRunningTime="2026-08-04 18:07:20.217662448 +0000 UTC m=+453.798881025"
Aug 04 18:07:21.194806 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:07:21.194773    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-ccbdf686-h9lf5"
Aug 04 18:07:27.228394 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:07:27.228355    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-tjxk2" event={"ID":"e88b512e-53a6-4cbd-9349-60d49c02f10e","Type":"ContainerStarted","Data":"19696da0ef562e23f30641e45089fa81fb53820df209c6abb7a77ac74a3b08cc"}
Aug 04 18:07:27.246164 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:07:27.246118    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-tjxk2" podStartSLOduration=1.649037242 podStartE2EDuration="8.246104382s" totalImagesPullingTime="6.59706714s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:07:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:07:27.243879419 +0000 UTC m=+460.825097994" watchObservedRunningTime="2026-08-04 18:07:27.246104382 +0000 UTC m=+460.827322960"
Aug 04 18:07:28.175337 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:07:28.175305    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-gr7sm"
Aug 04 18:07:30.190730 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:07:30.190696    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-56fb6f6856-m89z5"
Aug 04 18:07:32.201601 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:07:32.201569    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-ccbdf686-h9lf5"
Aug 04 18:08:02.114421 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:08:02.114387    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-b54df94b4-4r92k"]
Aug 04 18:08:02.140929 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:08:02.140897    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-b54df94b4-4r92k"]
Aug 04 18:08:02.141071 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:08:02.141010    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b54df94b4-4r92k"
Aug 04 18:08:02.144103 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:08:02.144080    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-n9vxf\""
Aug 04 18:08:02.260323 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:08:02.260286    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-c548b5d7d-gjpzh"]
Aug 04 18:08:02.263583 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:08:02.263565    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-gjpzh"
Aug 04 18:08:02.266027 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:08:02.265999    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jngwx\" (UniqueName: \"kubernetes.io/projected/887b91cf-272c-444d-8fda-0bcfcb4ad592-kube-api-access-jngwx\") pod \"authorino-b54df94b4-4r92k\" (UID: \"887b91cf-272c-444d-8fda-0bcfcb4ad592\") " pod="kuadrant-system/authorino-b54df94b4-4r92k"
Aug 04 18:08:02.280899 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:08:02.280872    2583 status_manager.go:1018] "Failed to get status for pod" err="pods \"authorino-c548b5d7d-gjpzh\" is forbidden: User \"system:node:ip-10-0-135-124.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-124.ec2.internal' and this object" podUID="ce29e332-8c3e-4957-83e1-20021fafcb2a" pod="kuadrant-system/authorino-c548b5d7d-gjpzh"
Aug 04 18:08:02.366569 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:08:02.366495    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jngwx\" (UniqueName: \"kubernetes.io/projected/887b91cf-272c-444d-8fda-0bcfcb4ad592-kube-api-access-jngwx\") pod \"authorino-b54df94b4-4r92k\" (UID: \"887b91cf-272c-444d-8fda-0bcfcb4ad592\") " pod="kuadrant-system/authorino-b54df94b4-4r92k"
Aug 04 18:08:02.366684 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:08:02.366577    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbfxh\" (UniqueName: \"kubernetes.io/projected/ce29e332-8c3e-4957-83e1-20021fafcb2a-kube-api-access-vbfxh\") pod \"authorino-c548b5d7d-gjpzh\" (UID: \"ce29e332-8c3e-4957-83e1-20021fafcb2a\") " pod="kuadrant-system/authorino-c548b5d7d-gjpzh"
Aug 04 18:08:02.369783 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:08:02.369761    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-gjpzh"]
Aug 04 18:08:02.398391 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:08:02.398361    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jngwx\" (UniqueName: \"kubernetes.io/projected/887b91cf-272c-444d-8fda-0bcfcb4ad592-kube-api-access-jngwx\") pod \"authorino-b54df94b4-4r92k\" (UID: \"887b91cf-272c-444d-8fda-0bcfcb4ad592\") " pod="kuadrant-system/authorino-b54df94b4-4r92k"
Aug 04 18:08:02.452141 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:08:02.452117    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b54df94b4-4r92k"
Aug 04 18:08:02.467399 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:08:02.467373    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbfxh\" (UniqueName: \"kubernetes.io/projected/ce29e332-8c3e-4957-83e1-20021fafcb2a-kube-api-access-vbfxh\") pod \"authorino-c548b5d7d-gjpzh\" (UID: \"ce29e332-8c3e-4957-83e1-20021fafcb2a\") " pod="kuadrant-system/authorino-c548b5d7d-gjpzh"
Aug 04 18:08:02.487537 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:08:02.487506    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbfxh\" (UniqueName: \"kubernetes.io/projected/ce29e332-8c3e-4957-83e1-20021fafcb2a-kube-api-access-vbfxh\") pod \"authorino-c548b5d7d-gjpzh\" (UID: \"ce29e332-8c3e-4957-83e1-20021fafcb2a\") " pod="kuadrant-system/authorino-c548b5d7d-gjpzh"
Aug 04 18:08:02.572270 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:08:02.572238    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-gjpzh"
Aug 04 18:08:02.598309 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:08:02.598283    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-b54df94b4-4r92k"]
Aug 04 18:08:02.598904 ip-10-0-135-124 kubenswrapper[2583]: W0804 18:08:02.598874    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod887b91cf_272c_444d_8fda_0bcfcb4ad592.slice/crio-7afbd087e8b6acb6dbca6f5ca26c4e37862507f1d511bf350179f536ff1686ca WatchSource:0}: Error finding container 7afbd087e8b6acb6dbca6f5ca26c4e37862507f1d511bf350179f536ff1686ca: Status 404 returned error can't find the container with id 7afbd087e8b6acb6dbca6f5ca26c4e37862507f1d511bf350179f536ff1686ca
Aug 04 18:08:02.719134 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:08:02.719100    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-gjpzh"]
Aug 04 18:08:02.722367 ip-10-0-135-124 kubenswrapper[2583]: W0804 18:08:02.722343    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podce29e332_8c3e_4957_83e1_20021fafcb2a.slice/crio-d24f4dfea611a4f3c4bb508c3befb0091b9813fde46e66c08c1f50ab913aff5d WatchSource:0}: Error finding container d24f4dfea611a4f3c4bb508c3befb0091b9813fde46e66c08c1f50ab913aff5d: Status 404 returned error can't find the container with id d24f4dfea611a4f3c4bb508c3befb0091b9813fde46e66c08c1f50ab913aff5d
Aug 04 18:08:03.411924 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:08:03.411885    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b54df94b4-4r92k" event={"ID":"887b91cf-272c-444d-8fda-0bcfcb4ad592","Type":"ContainerStarted","Data":"7afbd087e8b6acb6dbca6f5ca26c4e37862507f1d511bf350179f536ff1686ca"}
Aug 04 18:08:03.413047 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:08:03.413024    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-gjpzh" event={"ID":"ce29e332-8c3e-4957-83e1-20021fafcb2a","Type":"ContainerStarted","Data":"d24f4dfea611a4f3c4bb508c3befb0091b9813fde46e66c08c1f50ab913aff5d"}
Aug 04 18:08:07.437562 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:08:07.437527    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b54df94b4-4r92k" event={"ID":"887b91cf-272c-444d-8fda-0bcfcb4ad592","Type":"ContainerStarted","Data":"9f5fb749558db5c03021411603102c740cfdce9436906e1bca47b0ea178d5708"}
Aug 04 18:08:07.439128 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:08:07.439103    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-gjpzh" event={"ID":"ce29e332-8c3e-4957-83e1-20021fafcb2a","Type":"ContainerStarted","Data":"43474066e26a463ea1957f5c6d4dc5a78c8da6c0250b282e7b219dca47df4978"}
Aug 04 18:08:07.454980 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:08:07.454937    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-b54df94b4-4r92k" podStartSLOduration=1.1750199829999999 podStartE2EDuration="5.454921436s" totalImagesPullingTime="4.279901453s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:08:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:08:07.453530826 +0000 UTC m=+501.034749403" watchObservedRunningTime="2026-08-04 18:08:07.454921436 +0000 UTC m=+501.036140014"
Aug 04 18:08:07.470289 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:08:07.470244    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-c548b5d7d-gjpzh" podStartSLOduration=1.203814374 podStartE2EDuration="5.470229544s" totalImagesPullingTime="4.26641517s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:08:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:08:07.468570789 +0000 UTC m=+501.049789368" watchObservedRunningTime="2026-08-04 18:08:07.470229544 +0000 UTC m=+501.051448121"
Aug 04 18:08:07.495725 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:08:07.495699    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-b54df94b4-4r92k"]
Aug 04 18:08:09.449790 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:08:09.449748    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-b54df94b4-4r92k" podUID="887b91cf-272c-444d-8fda-0bcfcb4ad592" containerName="authorino" containerID="cri-o://9f5fb749558db5c03021411603102c740cfdce9436906e1bca47b0ea178d5708" gracePeriod=30
Aug 04 18:08:09.693633 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:08:09.693603    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b54df94b4-4r92k"
Aug 04 18:08:09.837614 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:08:09.837539    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jngwx\" (UniqueName: \"kubernetes.io/projected/887b91cf-272c-444d-8fda-0bcfcb4ad592-kube-api-access-jngwx\") pod \"887b91cf-272c-444d-8fda-0bcfcb4ad592\" (UID: \"887b91cf-272c-444d-8fda-0bcfcb4ad592\") "
Aug 04 18:08:09.839658 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:08:09.839633    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/887b91cf-272c-444d-8fda-0bcfcb4ad592-kube-api-access-jngwx" (OuterVolumeSpecName: "kube-api-access-jngwx") pod "887b91cf-272c-444d-8fda-0bcfcb4ad592" (UID: "887b91cf-272c-444d-8fda-0bcfcb4ad592"). InnerVolumeSpecName "kube-api-access-jngwx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:08:09.939057 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:08:09.939031    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jngwx\" (UniqueName: \"kubernetes.io/projected/887b91cf-272c-444d-8fda-0bcfcb4ad592-kube-api-access-jngwx\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:08:10.455906 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:08:10.455870    2583 generic.go:356] "Generic (PLEG): container finished" podID="887b91cf-272c-444d-8fda-0bcfcb4ad592" containerID="9f5fb749558db5c03021411603102c740cfdce9436906e1bca47b0ea178d5708" exitCode=0
Aug 04 18:08:10.456300 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:08:10.455909    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b54df94b4-4r92k"
Aug 04 18:08:10.456300 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:08:10.455950    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b54df94b4-4r92k" event={"ID":"887b91cf-272c-444d-8fda-0bcfcb4ad592","Type":"ContainerDied","Data":"9f5fb749558db5c03021411603102c740cfdce9436906e1bca47b0ea178d5708"}
Aug 04 18:08:10.456300 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:08:10.455994    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b54df94b4-4r92k" event={"ID":"887b91cf-272c-444d-8fda-0bcfcb4ad592","Type":"ContainerDied","Data":"7afbd087e8b6acb6dbca6f5ca26c4e37862507f1d511bf350179f536ff1686ca"}
Aug 04 18:08:10.456300 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:08:10.456012    2583 scope.go:117] "RemoveContainer" containerID="9f5fb749558db5c03021411603102c740cfdce9436906e1bca47b0ea178d5708"
Aug 04 18:08:10.468523 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:08:10.468505    2583 scope.go:117] "RemoveContainer" containerID="9f5fb749558db5c03021411603102c740cfdce9436906e1bca47b0ea178d5708"
Aug 04 18:08:10.468838 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:08:10.468820    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f5fb749558db5c03021411603102c740cfdce9436906e1bca47b0ea178d5708\": container with ID starting with 9f5fb749558db5c03021411603102c740cfdce9436906e1bca47b0ea178d5708 not found: ID does not exist" containerID="9f5fb749558db5c03021411603102c740cfdce9436906e1bca47b0ea178d5708"
Aug 04 18:08:10.468889 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:08:10.468847    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f5fb749558db5c03021411603102c740cfdce9436906e1bca47b0ea178d5708"} err="failed to get container status \"9f5fb749558db5c03021411603102c740cfdce9436906e1bca47b0ea178d5708\": rpc error: code = NotFound desc = could not find container \"9f5fb749558db5c03021411603102c740cfdce9436906e1bca47b0ea178d5708\": container with ID starting with 9f5fb749558db5c03021411603102c740cfdce9436906e1bca47b0ea178d5708 not found: ID does not exist"
Aug 04 18:08:10.483976 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:08:10.483952    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-b54df94b4-4r92k"]
Aug 04 18:08:10.487936 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:08:10.487916    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-b54df94b4-4r92k"]
Aug 04 18:08:10.942275 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:08:10.942244    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="887b91cf-272c-444d-8fda-0bcfcb4ad592" path="/var/lib/kubelet/pods/887b91cf-272c-444d-8fda-0bcfcb4ad592/volumes"
Aug 04 18:08:26.806358 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:08:26.806328    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-gjpzh"]
Aug 04 18:08:26.806778 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:08:26.806519    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-c548b5d7d-gjpzh" podUID="ce29e332-8c3e-4957-83e1-20021fafcb2a" containerName="authorino" containerID="cri-o://43474066e26a463ea1957f5c6d4dc5a78c8da6c0250b282e7b219dca47df4978" gracePeriod=30
Aug 04 18:08:27.050269 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:08:27.050242    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-gjpzh"
Aug 04 18:08:27.068993 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:08:27.068941    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vbfxh\" (UniqueName: \"kubernetes.io/projected/ce29e332-8c3e-4957-83e1-20021fafcb2a-kube-api-access-vbfxh\") pod \"ce29e332-8c3e-4957-83e1-20021fafcb2a\" (UID: \"ce29e332-8c3e-4957-83e1-20021fafcb2a\") "
Aug 04 18:08:27.071086 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:08:27.071051    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce29e332-8c3e-4957-83e1-20021fafcb2a-kube-api-access-vbfxh" (OuterVolumeSpecName: "kube-api-access-vbfxh") pod "ce29e332-8c3e-4957-83e1-20021fafcb2a" (UID: "ce29e332-8c3e-4957-83e1-20021fafcb2a"). InnerVolumeSpecName "kube-api-access-vbfxh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:08:27.169866 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:08:27.169839    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vbfxh\" (UniqueName: \"kubernetes.io/projected/ce29e332-8c3e-4957-83e1-20021fafcb2a-kube-api-access-vbfxh\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:08:27.547721 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:08:27.547690    2583 generic.go:356] "Generic (PLEG): container finished" podID="ce29e332-8c3e-4957-83e1-20021fafcb2a" containerID="43474066e26a463ea1957f5c6d4dc5a78c8da6c0250b282e7b219dca47df4978" exitCode=0
Aug 04 18:08:27.547872 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:08:27.547726    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-gjpzh"
Aug 04 18:08:27.547872 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:08:27.547755    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-gjpzh" event={"ID":"ce29e332-8c3e-4957-83e1-20021fafcb2a","Type":"ContainerDied","Data":"43474066e26a463ea1957f5c6d4dc5a78c8da6c0250b282e7b219dca47df4978"}
Aug 04 18:08:27.547872 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:08:27.547795    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-gjpzh" event={"ID":"ce29e332-8c3e-4957-83e1-20021fafcb2a","Type":"ContainerDied","Data":"d24f4dfea611a4f3c4bb508c3befb0091b9813fde46e66c08c1f50ab913aff5d"}
Aug 04 18:08:27.547872 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:08:27.547812    2583 scope.go:117] "RemoveContainer" containerID="43474066e26a463ea1957f5c6d4dc5a78c8da6c0250b282e7b219dca47df4978"
Aug 04 18:08:27.557391 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:08:27.557373    2583 scope.go:117] "RemoveContainer" containerID="43474066e26a463ea1957f5c6d4dc5a78c8da6c0250b282e7b219dca47df4978"
Aug 04 18:08:27.557666 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:08:27.557646    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43474066e26a463ea1957f5c6d4dc5a78c8da6c0250b282e7b219dca47df4978\": container with ID starting with 43474066e26a463ea1957f5c6d4dc5a78c8da6c0250b282e7b219dca47df4978 not found: ID does not exist" containerID="43474066e26a463ea1957f5c6d4dc5a78c8da6c0250b282e7b219dca47df4978"
Aug 04 18:08:27.557712 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:08:27.557675    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43474066e26a463ea1957f5c6d4dc5a78c8da6c0250b282e7b219dca47df4978"} err="failed to get container status \"43474066e26a463ea1957f5c6d4dc5a78c8da6c0250b282e7b219dca47df4978\": rpc error: code = NotFound desc = could not find container \"43474066e26a463ea1957f5c6d4dc5a78c8da6c0250b282e7b219dca47df4978\": container with ID starting with 43474066e26a463ea1957f5c6d4dc5a78c8da6c0250b282e7b219dca47df4978 not found: ID does not exist"
Aug 04 18:08:27.571813 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:08:27.571793    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-gjpzh"]
Aug 04 18:08:27.577026 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:08:27.577004    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-gjpzh"]
Aug 04 18:08:28.941486 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:08:28.941445    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ce29e332-8c3e-4957-83e1-20021fafcb2a" path="/var/lib/kubelet/pods/ce29e332-8c3e-4957-83e1-20021fafcb2a/volumes"
Aug 04 18:08:43.589650 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:08:43.589602    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-v894s"]
Aug 04 18:08:43.590120 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:08:43.589937    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce29e332-8c3e-4957-83e1-20021fafcb2a" containerName="authorino"
Aug 04 18:08:43.590120 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:08:43.589949    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce29e332-8c3e-4957-83e1-20021fafcb2a" containerName="authorino"
Aug 04 18:08:43.590120 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:08:43.589967    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="887b91cf-272c-444d-8fda-0bcfcb4ad592" containerName="authorino"
Aug 04 18:08:43.590120 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:08:43.589973    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="887b91cf-272c-444d-8fda-0bcfcb4ad592" containerName="authorino"
Aug 04 18:08:43.590120 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:08:43.590035    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="ce29e332-8c3e-4957-83e1-20021fafcb2a" containerName="authorino"
Aug 04 18:08:43.590120 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:08:43.590045    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="887b91cf-272c-444d-8fda-0bcfcb4ad592" containerName="authorino"
Aug 04 18:08:43.592956 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:08:43.592934    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-v894s"
Aug 04 18:08:43.605610 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:08:43.605588    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-v894s"]
Aug 04 18:08:43.707464 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:08:43.707431    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ac2a4755-e17f-423b-a44d-0dc532e255f9-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-v894s\" (UID: \"ac2a4755-e17f-423b-a44d-0dc532e255f9\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-v894s"
Aug 04 18:08:43.707667 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:08:43.707499    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/ac2a4755-e17f-423b-a44d-0dc532e255f9-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-v894s\" (UID: \"ac2a4755-e17f-423b-a44d-0dc532e255f9\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-v894s"
Aug 04 18:08:43.707667 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:08:43.707570    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/ac2a4755-e17f-423b-a44d-0dc532e255f9-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-v894s\" (UID: \"ac2a4755-e17f-423b-a44d-0dc532e255f9\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-v894s"
Aug 04 18:08:43.707667 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:08:43.707617    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9whr\" (UniqueName: \"kubernetes.io/projected/ac2a4755-e17f-423b-a44d-0dc532e255f9-kube-api-access-g9whr\") pod \"istiod-openshift-gateway-5677ccb49b-v894s\" (UID: \"ac2a4755-e17f-423b-a44d-0dc532e255f9\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-v894s"
Aug 04 18:08:43.707667 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:08:43.707639    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac2a4755-e17f-423b-a44d-0dc532e255f9-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-v894s\" (UID: \"ac2a4755-e17f-423b-a44d-0dc532e255f9\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-v894s"
Aug 04 18:08:43.707861 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:08:43.707688    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/ac2a4755-e17f-423b-a44d-0dc532e255f9-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-v894s\" (UID: \"ac2a4755-e17f-423b-a44d-0dc532e255f9\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-v894s"
Aug 04 18:08:43.707861 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:08:43.707735    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac2a4755-e17f-423b-a44d-0dc532e255f9-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-v894s\" (UID: \"ac2a4755-e17f-423b-a44d-0dc532e255f9\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-v894s"
Aug 04 18:08:43.808690 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:08:43.808654    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/ac2a4755-e17f-423b-a44d-0dc532e255f9-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-v894s\" (UID: \"ac2a4755-e17f-423b-a44d-0dc532e255f9\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-v894s"
Aug 04 18:08:43.808890 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:08:43.808702    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9whr\" (UniqueName: \"kubernetes.io/projected/ac2a4755-e17f-423b-a44d-0dc532e255f9-kube-api-access-g9whr\") pod \"istiod-openshift-gateway-5677ccb49b-v894s\" (UID: \"ac2a4755-e17f-423b-a44d-0dc532e255f9\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-v894s"
Aug 04 18:08:43.808890 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:08:43.808723    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac2a4755-e17f-423b-a44d-0dc532e255f9-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-v894s\" (UID: \"ac2a4755-e17f-423b-a44d-0dc532e255f9\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-v894s"
Aug 04 18:08:43.808890 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:08:43.808836    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/ac2a4755-e17f-423b-a44d-0dc532e255f9-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-v894s\" (UID: \"ac2a4755-e17f-423b-a44d-0dc532e255f9\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-v894s"
Aug 04 18:08:43.808890 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:08:43.808869    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac2a4755-e17f-423b-a44d-0dc532e255f9-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-v894s\" (UID: \"ac2a4755-e17f-423b-a44d-0dc532e255f9\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-v894s"
Aug 04 18:08:43.809107 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:08:43.808895    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ac2a4755-e17f-423b-a44d-0dc532e255f9-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-v894s\" (UID: \"ac2a4755-e17f-423b-a44d-0dc532e255f9\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-v894s"
Aug 04 18:08:43.809107 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:08:43.808948    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/ac2a4755-e17f-423b-a44d-0dc532e255f9-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-v894s\" (UID: \"ac2a4755-e17f-423b-a44d-0dc532e255f9\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-v894s"
Aug 04 18:08:43.809541 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:08:43.809508    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/ac2a4755-e17f-423b-a44d-0dc532e255f9-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-v894s\" (UID: \"ac2a4755-e17f-423b-a44d-0dc532e255f9\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-v894s"
Aug 04 18:08:43.811316 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:08:43.811287    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ac2a4755-e17f-423b-a44d-0dc532e255f9-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-v894s\" (UID: \"ac2a4755-e17f-423b-a44d-0dc532e255f9\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-v894s"
Aug 04 18:08:43.811431 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:08:43.811315    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/ac2a4755-e17f-423b-a44d-0dc532e255f9-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-v894s\" (UID: \"ac2a4755-e17f-423b-a44d-0dc532e255f9\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-v894s"
Aug 04 18:08:43.811431 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:08:43.811398    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac2a4755-e17f-423b-a44d-0dc532e255f9-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-v894s\" (UID: \"ac2a4755-e17f-423b-a44d-0dc532e255f9\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-v894s"
Aug 04 18:08:43.811615 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:08:43.811594    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/ac2a4755-e17f-423b-a44d-0dc532e255f9-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-v894s\" (UID: \"ac2a4755-e17f-423b-a44d-0dc532e255f9\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-v894s"
Aug 04 18:08:43.817269 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:08:43.817250    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac2a4755-e17f-423b-a44d-0dc532e255f9-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-v894s\" (UID: \"ac2a4755-e17f-423b-a44d-0dc532e255f9\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-v894s"
Aug 04 18:08:43.817890 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:08:43.817869    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9whr\" (UniqueName: \"kubernetes.io/projected/ac2a4755-e17f-423b-a44d-0dc532e255f9-kube-api-access-g9whr\") pod \"istiod-openshift-gateway-5677ccb49b-v894s\" (UID: \"ac2a4755-e17f-423b-a44d-0dc532e255f9\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-v894s"
Aug 04 18:08:43.902017 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:08:43.901925    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-v894s"
Aug 04 18:08:44.039261 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:08:44.039231    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-v894s"]
Aug 04 18:08:44.040643 ip-10-0-135-124 kubenswrapper[2583]: W0804 18:08:44.040610    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac2a4755_e17f_423b_a44d_0dc532e255f9.slice/crio-d09551aae4602a4262234b5bc24c5be43d7bae2b1d363291e46ed063992488b7 WatchSource:0}: Error finding container d09551aae4602a4262234b5bc24c5be43d7bae2b1d363291e46ed063992488b7: Status 404 returned error can't find the container with id d09551aae4602a4262234b5bc24c5be43d7bae2b1d363291e46ed063992488b7
Aug 04 18:08:44.042792 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:08:44.042754    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Aug 04 18:08:44.042862 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:08:44.042835    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Aug 04 18:08:44.664830 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:08:44.664793    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-v894s" event={"ID":"ac2a4755-e17f-423b-a44d-0dc532e255f9","Type":"ContainerStarted","Data":"0d80a0c954791d4d5868475478d642b72f1c3d9401247aa985a04c652a976305"}
Aug 04 18:08:44.665223 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:08:44.664867    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-v894s" event={"ID":"ac2a4755-e17f-423b-a44d-0dc532e255f9","Type":"ContainerStarted","Data":"d09551aae4602a4262234b5bc24c5be43d7bae2b1d363291e46ed063992488b7"}
Aug 04 18:08:44.665223 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:08:44.664900    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-v894s"
Aug 04 18:08:44.692727 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:08:44.692674    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-v894s" podStartSLOduration=1.69265732 podStartE2EDuration="1.69265732s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:08:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:08:44.689600476 +0000 UTC m=+538.270819066" watchObservedRunningTime="2026-08-04 18:08:44.69265732 +0000 UTC m=+538.273875898"
Aug 04 18:08:45.672030 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:08:45.672001    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-v894s"
Aug 04 18:08:45.730067 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:08:45.730033    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lvpcs"]
Aug 04 18:08:45.730297 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:08:45.730265    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lvpcs" podUID="8e649f5f-ed69-42cf-b451-4755a5243ae7" containerName="discovery" containerID="cri-o://d0b8c61d9624e617d3986ba16e2cc903ed83f1a2617d0030d2637ae7674e5417" gracePeriod=30
Aug 04 18:08:45.993508 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:08:45.993479    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lvpcs"
Aug 04 18:08:46.131008 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:08:46.130969    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/8e649f5f-ed69-42cf-b451-4755a5243ae7-istio-csr-ca-configmap\") pod \"8e649f5f-ed69-42cf-b451-4755a5243ae7\" (UID: \"8e649f5f-ed69-42cf-b451-4755a5243ae7\") "
Aug 04 18:08:46.131195 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:08:46.131048    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e649f5f-ed69-42cf-b451-4755a5243ae7-istio-token\") pod \"8e649f5f-ed69-42cf-b451-4755a5243ae7\" (UID: \"8e649f5f-ed69-42cf-b451-4755a5243ae7\") "
Aug 04 18:08:46.131195 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:08:46.131073    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e649f5f-ed69-42cf-b451-4755a5243ae7-local-certs\") pod \"8e649f5f-ed69-42cf-b451-4755a5243ae7\" (UID: \"8e649f5f-ed69-42cf-b451-4755a5243ae7\") "
Aug 04 18:08:46.131195 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:08:46.131100    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/8e649f5f-ed69-42cf-b451-4755a5243ae7-istio-csr-dns-cert\") pod \"8e649f5f-ed69-42cf-b451-4755a5243ae7\" (UID: \"8e649f5f-ed69-42cf-b451-4755a5243ae7\") "
Aug 04 18:08:46.131195 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:08:46.131143    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/8e649f5f-ed69-42cf-b451-4755a5243ae7-cacerts\") pod \"8e649f5f-ed69-42cf-b451-4755a5243ae7\" (UID: \"8e649f5f-ed69-42cf-b451-4755a5243ae7\") "
Aug 04 18:08:46.131195 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:08:46.131194    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6l6rj\" (UniqueName: \"kubernetes.io/projected/8e649f5f-ed69-42cf-b451-4755a5243ae7-kube-api-access-6l6rj\") pod \"8e649f5f-ed69-42cf-b451-4755a5243ae7\" (UID: \"8e649f5f-ed69-42cf-b451-4755a5243ae7\") "
Aug 04 18:08:46.131454 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:08:46.131223    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8e649f5f-ed69-42cf-b451-4755a5243ae7-istio-kubeconfig\") pod \"8e649f5f-ed69-42cf-b451-4755a5243ae7\" (UID: \"8e649f5f-ed69-42cf-b451-4755a5243ae7\") "
Aug 04 18:08:46.131542 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:08:46.131499    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e649f5f-ed69-42cf-b451-4755a5243ae7-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "8e649f5f-ed69-42cf-b451-4755a5243ae7" (UID: "8e649f5f-ed69-42cf-b451-4755a5243ae7"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:08:46.133728 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:08:46.133700    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e649f5f-ed69-42cf-b451-4755a5243ae7-cacerts" (OuterVolumeSpecName: "cacerts") pod "8e649f5f-ed69-42cf-b451-4755a5243ae7" (UID: "8e649f5f-ed69-42cf-b451-4755a5243ae7"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:08:46.133844 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:08:46.133760    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e649f5f-ed69-42cf-b451-4755a5243ae7-kube-api-access-6l6rj" (OuterVolumeSpecName: "kube-api-access-6l6rj") pod "8e649f5f-ed69-42cf-b451-4755a5243ae7" (UID: "8e649f5f-ed69-42cf-b451-4755a5243ae7"). InnerVolumeSpecName "kube-api-access-6l6rj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:08:46.133924 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:08:46.133868    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e649f5f-ed69-42cf-b451-4755a5243ae7-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "8e649f5f-ed69-42cf-b451-4755a5243ae7" (UID: "8e649f5f-ed69-42cf-b451-4755a5243ae7"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:08:46.133924 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:08:46.133894    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e649f5f-ed69-42cf-b451-4755a5243ae7-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "8e649f5f-ed69-42cf-b451-4755a5243ae7" (UID: "8e649f5f-ed69-42cf-b451-4755a5243ae7"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:08:46.134035 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:08:46.133933    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e649f5f-ed69-42cf-b451-4755a5243ae7-local-certs" (OuterVolumeSpecName: "local-certs") pod "8e649f5f-ed69-42cf-b451-4755a5243ae7" (UID: "8e649f5f-ed69-42cf-b451-4755a5243ae7"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:08:46.134035 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:08:46.134013    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e649f5f-ed69-42cf-b451-4755a5243ae7-istio-token" (OuterVolumeSpecName: "istio-token") pod "8e649f5f-ed69-42cf-b451-4755a5243ae7" (UID: "8e649f5f-ed69-42cf-b451-4755a5243ae7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:08:46.232297 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:08:46.232217    2583 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/8e649f5f-ed69-42cf-b451-4755a5243ae7-cacerts\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:08:46.232297 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:08:46.232244    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6l6rj\" (UniqueName: \"kubernetes.io/projected/8e649f5f-ed69-42cf-b451-4755a5243ae7-kube-api-access-6l6rj\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:08:46.232297 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:08:46.232256    2583 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8e649f5f-ed69-42cf-b451-4755a5243ae7-istio-kubeconfig\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:08:46.232297 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:08:46.232265    2583 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/8e649f5f-ed69-42cf-b451-4755a5243ae7-istio-csr-ca-configmap\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:08:46.232297 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:08:46.232273    2583 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e649f5f-ed69-42cf-b451-4755a5243ae7-istio-token\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:08:46.232297 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:08:46.232282    2583 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e649f5f-ed69-42cf-b451-4755a5243ae7-local-certs\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:08:46.232297 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:08:46.232290    2583 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/8e649f5f-ed69-42cf-b451-4755a5243ae7-istio-csr-dns-cert\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:08:46.676556 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:08:46.676450    2583 generic.go:356] "Generic (PLEG): container finished" podID="8e649f5f-ed69-42cf-b451-4755a5243ae7" containerID="d0b8c61d9624e617d3986ba16e2cc903ed83f1a2617d0030d2637ae7674e5417" exitCode=0
Aug 04 18:08:46.676556 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:08:46.676496    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lvpcs" event={"ID":"8e649f5f-ed69-42cf-b451-4755a5243ae7","Type":"ContainerDied","Data":"d0b8c61d9624e617d3986ba16e2cc903ed83f1a2617d0030d2637ae7674e5417"}
Aug 04 18:08:46.676556 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:08:46.676511    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lvpcs"
Aug 04 18:08:46.676556 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:08:46.676544    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lvpcs" event={"ID":"8e649f5f-ed69-42cf-b451-4755a5243ae7","Type":"ContainerDied","Data":"d16b6a83837cfda04265a09c3321fbd5aac11ae264ac5eee17c16307e91406fb"}
Aug 04 18:08:46.677177 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:08:46.676563    2583 scope.go:117] "RemoveContainer" containerID="d0b8c61d9624e617d3986ba16e2cc903ed83f1a2617d0030d2637ae7674e5417"
Aug 04 18:08:46.686410 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:08:46.686393    2583 scope.go:117] "RemoveContainer" containerID="d0b8c61d9624e617d3986ba16e2cc903ed83f1a2617d0030d2637ae7674e5417"
Aug 04 18:08:46.686707 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:08:46.686690    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0b8c61d9624e617d3986ba16e2cc903ed83f1a2617d0030d2637ae7674e5417\": container with ID starting with d0b8c61d9624e617d3986ba16e2cc903ed83f1a2617d0030d2637ae7674e5417 not found: ID does not exist" containerID="d0b8c61d9624e617d3986ba16e2cc903ed83f1a2617d0030d2637ae7674e5417"
Aug 04 18:08:46.686770 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:08:46.686720    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0b8c61d9624e617d3986ba16e2cc903ed83f1a2617d0030d2637ae7674e5417"} err="failed to get container status \"d0b8c61d9624e617d3986ba16e2cc903ed83f1a2617d0030d2637ae7674e5417\": rpc error: code = NotFound desc = could not find container \"d0b8c61d9624e617d3986ba16e2cc903ed83f1a2617d0030d2637ae7674e5417\": container with ID starting with d0b8c61d9624e617d3986ba16e2cc903ed83f1a2617d0030d2637ae7674e5417 not found: ID does not exist"
Aug 04 18:08:46.699927 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:08:46.699907    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lvpcs"]
Aug 04 18:08:46.707937 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:08:46.707917    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-lvpcs"]
Aug 04 18:08:46.941784 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:08:46.941756    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8e649f5f-ed69-42cf-b451-4755a5243ae7" path="/var/lib/kubelet/pods/8e649f5f-ed69-42cf-b451-4755a5243ae7/volumes"
Aug 04 18:08:55.161342 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:08:55.161307    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-5f9d7f69b6-2knls"]
Aug 04 18:08:55.161784 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:08:55.161653    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e649f5f-ed69-42cf-b451-4755a5243ae7" containerName="discovery"
Aug 04 18:08:55.161784 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:08:55.161665    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e649f5f-ed69-42cf-b451-4755a5243ae7" containerName="discovery"
Aug 04 18:08:55.161784 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:08:55.161717    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e649f5f-ed69-42cf-b451-4755a5243ae7" containerName="discovery"
Aug 04 18:08:55.166193 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:08:55.166175    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-5f9d7f69b6-2knls"
Aug 04 18:08:55.168830 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:08:55.168808    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Aug 04 18:08:55.168967 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:08:55.168842    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-webhook-server-cert\""
Aug 04 18:08:55.168967 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:08:55.168865    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Aug 04 18:08:55.169116 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:08:55.169102    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-controller-manager-dockercfg-wv9pj\""
Aug 04 18:08:55.174624 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:08:55.174603    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-5f9d7f69b6-2knls"]
Aug 04 18:08:55.210655 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:08:55.210630    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-64568bcd49-p22ln"]
Aug 04 18:08:55.213831 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:08:55.213815    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-p22ln"
Aug 04 18:08:55.216370 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:08:55.216351    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Aug 04 18:08:55.216491 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:08:55.216374    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-drfsn\""
Aug 04 18:08:55.221592 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:08:55.221571    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-p22ln"]
Aug 04 18:08:55.300809 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:08:55.300781    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9b0a26ad-bf1a-4d55-a17d-f64b63ae0e63-cert\") pod \"kserve-controller-manager-5f9d7f69b6-2knls\" (UID: \"9b0a26ad-bf1a-4d55-a17d-f64b63ae0e63\") " pod="kserve/kserve-controller-manager-5f9d7f69b6-2knls"
Aug 04 18:08:55.300955 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:08:55.300821    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhk2m\" (UniqueName: \"kubernetes.io/projected/e5c04e29-0a3f-4a2d-9809-9a62baa34378-kube-api-access-jhk2m\") pod \"seaweedfs-64568bcd49-p22ln\" (UID: \"e5c04e29-0a3f-4a2d-9809-9a62baa34378\") " pod="kserve/seaweedfs-64568bcd49-p22ln"
Aug 04 18:08:55.300955 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:08:55.300850    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cj4l9\" (UniqueName: \"kubernetes.io/projected/9b0a26ad-bf1a-4d55-a17d-f64b63ae0e63-kube-api-access-cj4l9\") pod \"kserve-controller-manager-5f9d7f69b6-2knls\" (UID: \"9b0a26ad-bf1a-4d55-a17d-f64b63ae0e63\") " pod="kserve/kserve-controller-manager-5f9d7f69b6-2knls"
Aug 04 18:08:55.300955 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:08:55.300915    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/e5c04e29-0a3f-4a2d-9809-9a62baa34378-data\") pod \"seaweedfs-64568bcd49-p22ln\" (UID: \"e5c04e29-0a3f-4a2d-9809-9a62baa34378\") " pod="kserve/seaweedfs-64568bcd49-p22ln"
Aug 04 18:08:55.401991 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:08:55.401959    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/e5c04e29-0a3f-4a2d-9809-9a62baa34378-data\") pod \"seaweedfs-64568bcd49-p22ln\" (UID: \"e5c04e29-0a3f-4a2d-9809-9a62baa34378\") " pod="kserve/seaweedfs-64568bcd49-p22ln"
Aug 04 18:08:55.402133 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:08:55.402030    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9b0a26ad-bf1a-4d55-a17d-f64b63ae0e63-cert\") pod \"kserve-controller-manager-5f9d7f69b6-2knls\" (UID: \"9b0a26ad-bf1a-4d55-a17d-f64b63ae0e63\") " pod="kserve/kserve-controller-manager-5f9d7f69b6-2knls"
Aug 04 18:08:55.402133 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:08:55.402064    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhk2m\" (UniqueName: \"kubernetes.io/projected/e5c04e29-0a3f-4a2d-9809-9a62baa34378-kube-api-access-jhk2m\") pod \"seaweedfs-64568bcd49-p22ln\" (UID: \"e5c04e29-0a3f-4a2d-9809-9a62baa34378\") " pod="kserve/seaweedfs-64568bcd49-p22ln"
Aug 04 18:08:55.402133 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:08:55.402092    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cj4l9\" (UniqueName: \"kubernetes.io/projected/9b0a26ad-bf1a-4d55-a17d-f64b63ae0e63-kube-api-access-cj4l9\") pod \"kserve-controller-manager-5f9d7f69b6-2knls\" (UID: \"9b0a26ad-bf1a-4d55-a17d-f64b63ae0e63\") " pod="kserve/kserve-controller-manager-5f9d7f69b6-2knls"
Aug 04 18:08:55.402390 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:08:55.402370    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/e5c04e29-0a3f-4a2d-9809-9a62baa34378-data\") pod \"seaweedfs-64568bcd49-p22ln\" (UID: \"e5c04e29-0a3f-4a2d-9809-9a62baa34378\") " pod="kserve/seaweedfs-64568bcd49-p22ln"
Aug 04 18:08:55.404561 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:08:55.404535    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9b0a26ad-bf1a-4d55-a17d-f64b63ae0e63-cert\") pod \"kserve-controller-manager-5f9d7f69b6-2knls\" (UID: \"9b0a26ad-bf1a-4d55-a17d-f64b63ae0e63\") " pod="kserve/kserve-controller-manager-5f9d7f69b6-2knls"
Aug 04 18:08:55.412880 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:08:55.412833    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhk2m\" (UniqueName: \"kubernetes.io/projected/e5c04e29-0a3f-4a2d-9809-9a62baa34378-kube-api-access-jhk2m\") pod \"seaweedfs-64568bcd49-p22ln\" (UID: \"e5c04e29-0a3f-4a2d-9809-9a62baa34378\") " pod="kserve/seaweedfs-64568bcd49-p22ln"
Aug 04 18:08:55.416146 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:08:55.416122    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cj4l9\" (UniqueName: \"kubernetes.io/projected/9b0a26ad-bf1a-4d55-a17d-f64b63ae0e63-kube-api-access-cj4l9\") pod \"kserve-controller-manager-5f9d7f69b6-2knls\" (UID: \"9b0a26ad-bf1a-4d55-a17d-f64b63ae0e63\") " pod="kserve/kserve-controller-manager-5f9d7f69b6-2knls"
Aug 04 18:08:55.477003 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:08:55.476972    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-5f9d7f69b6-2knls"
Aug 04 18:08:55.523760 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:08:55.523729    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-p22ln"
Aug 04 18:08:55.625078 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:08:55.625040    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-5f9d7f69b6-2knls"]
Aug 04 18:08:55.629254 ip-10-0-135-124 kubenswrapper[2583]: W0804 18:08:55.629209    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b0a26ad_bf1a_4d55_a17d_f64b63ae0e63.slice/crio-498d3c1177f999044fdc3f960abaa5625d0691c03364d7a43f7a5307b38b0ab8 WatchSource:0}: Error finding container 498d3c1177f999044fdc3f960abaa5625d0691c03364d7a43f7a5307b38b0ab8: Status 404 returned error can't find the container with id 498d3c1177f999044fdc3f960abaa5625d0691c03364d7a43f7a5307b38b0ab8
Aug 04 18:08:55.673954 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:08:55.673931    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-p22ln"]
Aug 04 18:08:55.675480 ip-10-0-135-124 kubenswrapper[2583]: W0804 18:08:55.675443    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5c04e29_0a3f_4a2d_9809_9a62baa34378.slice/crio-252bfe2469c30273fac3a64980c69e41d331daca1383dc43a581896b8efbec9e WatchSource:0}: Error finding container 252bfe2469c30273fac3a64980c69e41d331daca1383dc43a581896b8efbec9e: Status 404 returned error can't find the container with id 252bfe2469c30273fac3a64980c69e41d331daca1383dc43a581896b8efbec9e
Aug 04 18:08:55.720201 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:08:55.720173    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-5f9d7f69b6-2knls" event={"ID":"9b0a26ad-bf1a-4d55-a17d-f64b63ae0e63","Type":"ContainerStarted","Data":"498d3c1177f999044fdc3f960abaa5625d0691c03364d7a43f7a5307b38b0ab8"}
Aug 04 18:08:55.721299 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:08:55.721271    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-p22ln" event={"ID":"e5c04e29-0a3f-4a2d-9809-9a62baa34378","Type":"ContainerStarted","Data":"252bfe2469c30273fac3a64980c69e41d331daca1383dc43a581896b8efbec9e"}
Aug 04 18:08:59.748211 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:08:59.748178    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-5f9d7f69b6-2knls" event={"ID":"9b0a26ad-bf1a-4d55-a17d-f64b63ae0e63","Type":"ContainerStarted","Data":"f903ece8c373d7dbb5d2c85b1b11f232be116f283971526dbb4a939f075d9da0"}
Aug 04 18:08:59.748611 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:08:59.748255    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-5f9d7f69b6-2knls"
Aug 04 18:08:59.768207 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:08:59.768157    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-5f9d7f69b6-2knls" podStartSLOduration=0.782602856 podStartE2EDuration="4.76814084s" totalImagesPullingTime="3.985537984s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:08:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:08:59.767977486 +0000 UTC m=+553.349196066" watchObservedRunningTime="2026-08-04 18:08:59.76814084 +0000 UTC m=+553.349359418"
Aug 04 18:09:00.755047 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:09:00.755005    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-p22ln" event={"ID":"e5c04e29-0a3f-4a2d-9809-9a62baa34378","Type":"ContainerStarted","Data":"55c2d420e84197dcb6d7e79e7c3da520f206c92aec6d7a301a849630699c57c3"}
Aug 04 18:09:00.755461 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:09:00.755267    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-64568bcd49-p22ln"
Aug 04 18:09:00.772537 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:09:00.772494    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-64568bcd49-p22ln" podStartSLOduration=1.740988566 podStartE2EDuration="5.772461287s" totalImagesPullingTime="4.031472721s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:08:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:09:00.77168869 +0000 UTC m=+554.352907269" watchObservedRunningTime="2026-08-04 18:09:00.772461287 +0000 UTC m=+554.353679843"
Aug 04 18:09:06.762897 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:09:06.762865    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-64568bcd49-p22ln"
Aug 04 18:09:30.761002 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:09:30.760968    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-5f9d7f69b6-2knls"
Aug 04 18:09:38.999621 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:09:38.999584    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-5f9d7f69b6-2knls"]
Aug 04 18:09:39.000027 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:09:38.999783    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve/kserve-controller-manager-5f9d7f69b6-2knls" podUID="9b0a26ad-bf1a-4d55-a17d-f64b63ae0e63" containerName="manager" containerID="cri-o://f903ece8c373d7dbb5d2c85b1b11f232be116f283971526dbb4a939f075d9da0" gracePeriod=10
Aug 04 18:09:39.023395 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:09:39.023362    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-5f9d7f69b6-gmpfz"]
Aug 04 18:09:39.026111 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:09:39.026090    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-5f9d7f69b6-gmpfz"
Aug 04 18:09:39.037346 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:09:39.037320    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-5f9d7f69b6-gmpfz"]
Aug 04 18:09:39.061138 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:09:39.061113    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ft4q\" (UniqueName: \"kubernetes.io/projected/0522cb81-a78a-4bdc-a3c4-aef8c7ac3f24-kube-api-access-9ft4q\") pod \"kserve-controller-manager-5f9d7f69b6-gmpfz\" (UID: \"0522cb81-a78a-4bdc-a3c4-aef8c7ac3f24\") " pod="kserve/kserve-controller-manager-5f9d7f69b6-gmpfz"
Aug 04 18:09:39.061259 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:09:39.061163    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0522cb81-a78a-4bdc-a3c4-aef8c7ac3f24-cert\") pod \"kserve-controller-manager-5f9d7f69b6-gmpfz\" (UID: \"0522cb81-a78a-4bdc-a3c4-aef8c7ac3f24\") " pod="kserve/kserve-controller-manager-5f9d7f69b6-gmpfz"
Aug 04 18:09:39.162058 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:09:39.162022    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ft4q\" (UniqueName: \"kubernetes.io/projected/0522cb81-a78a-4bdc-a3c4-aef8c7ac3f24-kube-api-access-9ft4q\") pod \"kserve-controller-manager-5f9d7f69b6-gmpfz\" (UID: \"0522cb81-a78a-4bdc-a3c4-aef8c7ac3f24\") " pod="kserve/kserve-controller-manager-5f9d7f69b6-gmpfz"
Aug 04 18:09:39.162221 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:09:39.162078    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0522cb81-a78a-4bdc-a3c4-aef8c7ac3f24-cert\") pod \"kserve-controller-manager-5f9d7f69b6-gmpfz\" (UID: \"0522cb81-a78a-4bdc-a3c4-aef8c7ac3f24\") " pod="kserve/kserve-controller-manager-5f9d7f69b6-gmpfz"
Aug 04 18:09:39.164963 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:09:39.164772    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0522cb81-a78a-4bdc-a3c4-aef8c7ac3f24-cert\") pod \"kserve-controller-manager-5f9d7f69b6-gmpfz\" (UID: \"0522cb81-a78a-4bdc-a3c4-aef8c7ac3f24\") " pod="kserve/kserve-controller-manager-5f9d7f69b6-gmpfz"
Aug 04 18:09:39.172161 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:09:39.172133    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ft4q\" (UniqueName: \"kubernetes.io/projected/0522cb81-a78a-4bdc-a3c4-aef8c7ac3f24-kube-api-access-9ft4q\") pod \"kserve-controller-manager-5f9d7f69b6-gmpfz\" (UID: \"0522cb81-a78a-4bdc-a3c4-aef8c7ac3f24\") " pod="kserve/kserve-controller-manager-5f9d7f69b6-gmpfz"
Aug 04 18:09:39.244693 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:09:39.244670    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-5f9d7f69b6-2knls"
Aug 04 18:09:39.363746 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:09:39.363664    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cj4l9\" (UniqueName: \"kubernetes.io/projected/9b0a26ad-bf1a-4d55-a17d-f64b63ae0e63-kube-api-access-cj4l9\") pod \"9b0a26ad-bf1a-4d55-a17d-f64b63ae0e63\" (UID: \"9b0a26ad-bf1a-4d55-a17d-f64b63ae0e63\") "
Aug 04 18:09:39.363746 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:09:39.363718    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9b0a26ad-bf1a-4d55-a17d-f64b63ae0e63-cert\") pod \"9b0a26ad-bf1a-4d55-a17d-f64b63ae0e63\" (UID: \"9b0a26ad-bf1a-4d55-a17d-f64b63ae0e63\") "
Aug 04 18:09:39.365841 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:09:39.365810    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b0a26ad-bf1a-4d55-a17d-f64b63ae0e63-kube-api-access-cj4l9" (OuterVolumeSpecName: "kube-api-access-cj4l9") pod "9b0a26ad-bf1a-4d55-a17d-f64b63ae0e63" (UID: "9b0a26ad-bf1a-4d55-a17d-f64b63ae0e63"). InnerVolumeSpecName "kube-api-access-cj4l9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:09:39.365950 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:09:39.365860    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b0a26ad-bf1a-4d55-a17d-f64b63ae0e63-cert" (OuterVolumeSpecName: "cert") pod "9b0a26ad-bf1a-4d55-a17d-f64b63ae0e63" (UID: "9b0a26ad-bf1a-4d55-a17d-f64b63ae0e63"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:09:39.374037 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:09:39.374015    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-5f9d7f69b6-gmpfz"
Aug 04 18:09:39.465047 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:09:39.465011    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cj4l9\" (UniqueName: \"kubernetes.io/projected/9b0a26ad-bf1a-4d55-a17d-f64b63ae0e63-kube-api-access-cj4l9\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:09:39.465047 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:09:39.465046    2583 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9b0a26ad-bf1a-4d55-a17d-f64b63ae0e63-cert\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:09:39.502834 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:09:39.502811    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-5f9d7f69b6-gmpfz"]
Aug 04 18:09:39.504556 ip-10-0-135-124 kubenswrapper[2583]: W0804 18:09:39.504527    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0522cb81_a78a_4bdc_a3c4_aef8c7ac3f24.slice/crio-7463cf3b0135cd51fcb8e74601ef34894f3b1bafa66105017ea1ad73fbc2b64b WatchSource:0}: Error finding container 7463cf3b0135cd51fcb8e74601ef34894f3b1bafa66105017ea1ad73fbc2b64b: Status 404 returned error can't find the container with id 7463cf3b0135cd51fcb8e74601ef34894f3b1bafa66105017ea1ad73fbc2b64b
Aug 04 18:09:39.960323 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:09:39.960290    2583 generic.go:356] "Generic (PLEG): container finished" podID="9b0a26ad-bf1a-4d55-a17d-f64b63ae0e63" containerID="f903ece8c373d7dbb5d2c85b1b11f232be116f283971526dbb4a939f075d9da0" exitCode=0
Aug 04 18:09:39.960527 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:09:39.960330    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-5f9d7f69b6-2knls"
Aug 04 18:09:39.960527 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:09:39.960376    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-5f9d7f69b6-2knls" event={"ID":"9b0a26ad-bf1a-4d55-a17d-f64b63ae0e63","Type":"ContainerDied","Data":"f903ece8c373d7dbb5d2c85b1b11f232be116f283971526dbb4a939f075d9da0"}
Aug 04 18:09:39.960527 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:09:39.960426    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-5f9d7f69b6-2knls" event={"ID":"9b0a26ad-bf1a-4d55-a17d-f64b63ae0e63","Type":"ContainerDied","Data":"498d3c1177f999044fdc3f960abaa5625d0691c03364d7a43f7a5307b38b0ab8"}
Aug 04 18:09:39.960527 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:09:39.960457    2583 scope.go:117] "RemoveContainer" containerID="f903ece8c373d7dbb5d2c85b1b11f232be116f283971526dbb4a939f075d9da0"
Aug 04 18:09:39.961742 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:09:39.961716    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-5f9d7f69b6-gmpfz" event={"ID":"0522cb81-a78a-4bdc-a3c4-aef8c7ac3f24","Type":"ContainerStarted","Data":"7463cf3b0135cd51fcb8e74601ef34894f3b1bafa66105017ea1ad73fbc2b64b"}
Aug 04 18:09:39.969125 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:09:39.969107    2583 scope.go:117] "RemoveContainer" containerID="f903ece8c373d7dbb5d2c85b1b11f232be116f283971526dbb4a939f075d9da0"
Aug 04 18:09:39.969366 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:09:39.969348    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f903ece8c373d7dbb5d2c85b1b11f232be116f283971526dbb4a939f075d9da0\": container with ID starting with f903ece8c373d7dbb5d2c85b1b11f232be116f283971526dbb4a939f075d9da0 not found: ID does not exist" containerID="f903ece8c373d7dbb5d2c85b1b11f232be116f283971526dbb4a939f075d9da0"
Aug 04 18:09:39.969406 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:09:39.969374    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f903ece8c373d7dbb5d2c85b1b11f232be116f283971526dbb4a939f075d9da0"} err="failed to get container status \"f903ece8c373d7dbb5d2c85b1b11f232be116f283971526dbb4a939f075d9da0\": rpc error: code = NotFound desc = could not find container \"f903ece8c373d7dbb5d2c85b1b11f232be116f283971526dbb4a939f075d9da0\": container with ID starting with f903ece8c373d7dbb5d2c85b1b11f232be116f283971526dbb4a939f075d9da0 not found: ID does not exist"
Aug 04 18:09:39.981992 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:09:39.981966    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-5f9d7f69b6-2knls"]
Aug 04 18:09:39.987664 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:09:39.987644    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/kserve-controller-manager-5f9d7f69b6-2knls"]
Aug 04 18:09:40.942592 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:09:40.942560    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9b0a26ad-bf1a-4d55-a17d-f64b63ae0e63" path="/var/lib/kubelet/pods/9b0a26ad-bf1a-4d55-a17d-f64b63ae0e63/volumes"
Aug 04 18:09:40.968425 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:09:40.968395    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-5f9d7f69b6-gmpfz" event={"ID":"0522cb81-a78a-4bdc-a3c4-aef8c7ac3f24","Type":"ContainerStarted","Data":"8ceb39791e128e7649e82b0ba24a9d70c4f3176471b0aaa12d588ab669e5f07e"}
Aug 04 18:09:40.968602 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:09:40.968459    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-5f9d7f69b6-gmpfz"
Aug 04 18:09:40.988188 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:09:40.988141    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-5f9d7f69b6-gmpfz" podStartSLOduration=1.50069836 podStartE2EDuration="1.988126421s" totalImagesPullingTime="487.428061ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:09:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:09:40.985593897 +0000 UTC m=+594.566812474" watchObservedRunningTime="2026-08-04 18:09:40.988126421 +0000 UTC m=+594.569345027"
Aug 04 18:09:46.888887 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:09:46.888856    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-wskfb_6157c2d3-2fd0-4544-88ad-434ac75f8c04/console-operator/1.log"
Aug 04 18:09:46.889322 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:09:46.889039    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-wskfb_6157c2d3-2fd0-4544-88ad-434ac75f8c04/console-operator/1.log"
Aug 04 18:10:11.979297 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:10:11.979221    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-5f9d7f69b6-gmpfz"
Aug 04 18:10:13.592499 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:10:13.592455    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-7b954ffc95-dwm6d"]
Aug 04 18:10:13.593075 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:10:13.593051    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b0a26ad-bf1a-4d55-a17d-f64b63ae0e63" containerName="manager"
Aug 04 18:10:13.593157 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:10:13.593077    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b0a26ad-bf1a-4d55-a17d-f64b63ae0e63" containerName="manager"
Aug 04 18:10:13.593208 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:10:13.593164    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b0a26ad-bf1a-4d55-a17d-f64b63ae0e63" containerName="manager"
Aug 04 18:10:13.595031 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:10:13.595011    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-dwm6d"
Aug 04 18:10:13.597673 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:10:13.597650    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\""
Aug 04 18:10:13.597830 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:10:13.597815    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-jkm57\""
Aug 04 18:10:13.608169 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:10:13.608145    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-dwm6d"]
Aug 04 18:10:13.730736 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:10:13.730698    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lr2dj\" (UniqueName: \"kubernetes.io/projected/baa008cd-0d29-44f9-8cf8-aba20b62b76f-kube-api-access-lr2dj\") pod \"model-serving-api-7b954ffc95-dwm6d\" (UID: \"baa008cd-0d29-44f9-8cf8-aba20b62b76f\") " pod="kserve/model-serving-api-7b954ffc95-dwm6d"
Aug 04 18:10:13.730906 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:10:13.730746    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/baa008cd-0d29-44f9-8cf8-aba20b62b76f-tls-certs\") pod \"model-serving-api-7b954ffc95-dwm6d\" (UID: \"baa008cd-0d29-44f9-8cf8-aba20b62b76f\") " pod="kserve/model-serving-api-7b954ffc95-dwm6d"
Aug 04 18:10:13.831619 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:10:13.831582    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lr2dj\" (UniqueName: \"kubernetes.io/projected/baa008cd-0d29-44f9-8cf8-aba20b62b76f-kube-api-access-lr2dj\") pod \"model-serving-api-7b954ffc95-dwm6d\" (UID: \"baa008cd-0d29-44f9-8cf8-aba20b62b76f\") " pod="kserve/model-serving-api-7b954ffc95-dwm6d"
Aug 04 18:10:13.831774 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:10:13.831624    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/baa008cd-0d29-44f9-8cf8-aba20b62b76f-tls-certs\") pod \"model-serving-api-7b954ffc95-dwm6d\" (UID: \"baa008cd-0d29-44f9-8cf8-aba20b62b76f\") " pod="kserve/model-serving-api-7b954ffc95-dwm6d"
Aug 04 18:10:13.834032 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:10:13.834008    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/baa008cd-0d29-44f9-8cf8-aba20b62b76f-tls-certs\") pod \"model-serving-api-7b954ffc95-dwm6d\" (UID: \"baa008cd-0d29-44f9-8cf8-aba20b62b76f\") " pod="kserve/model-serving-api-7b954ffc95-dwm6d"
Aug 04 18:10:13.840732 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:10:13.840702    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lr2dj\" (UniqueName: \"kubernetes.io/projected/baa008cd-0d29-44f9-8cf8-aba20b62b76f-kube-api-access-lr2dj\") pod \"model-serving-api-7b954ffc95-dwm6d\" (UID: \"baa008cd-0d29-44f9-8cf8-aba20b62b76f\") " pod="kserve/model-serving-api-7b954ffc95-dwm6d"
Aug 04 18:10:13.905605 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:10:13.905551    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-dwm6d"
Aug 04 18:10:14.038358 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:10:14.038338    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-dwm6d"]
Aug 04 18:10:14.040807 ip-10-0-135-124 kubenswrapper[2583]: W0804 18:10:14.040780    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbaa008cd_0d29_44f9_8cf8_aba20b62b76f.slice/crio-4d6387ae0cfdb5eff8abdbed3e8b4a7ff9a7439da8b4dcf2519f9fe5388d17c7 WatchSource:0}: Error finding container 4d6387ae0cfdb5eff8abdbed3e8b4a7ff9a7439da8b4dcf2519f9fe5388d17c7: Status 404 returned error can't find the container with id 4d6387ae0cfdb5eff8abdbed3e8b4a7ff9a7439da8b4dcf2519f9fe5388d17c7
Aug 04 18:10:14.144196 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:10:14.144170    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-dwm6d" event={"ID":"baa008cd-0d29-44f9-8cf8-aba20b62b76f","Type":"ContainerStarted","Data":"4d6387ae0cfdb5eff8abdbed3e8b4a7ff9a7439da8b4dcf2519f9fe5388d17c7"}
Aug 04 18:10:16.158458 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:10:16.158428    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-dwm6d" event={"ID":"baa008cd-0d29-44f9-8cf8-aba20b62b76f","Type":"ContainerStarted","Data":"ed526caf4f7a72eb1e982e4677e658d23b03c562c47e603455a54af826ee821f"}
Aug 04 18:10:16.158808 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:10:16.158523    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-7b954ffc95-dwm6d"
Aug 04 18:10:16.180040 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:10:16.180002    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/model-serving-api-7b954ffc95-dwm6d" podStartSLOduration=1.174375618 podStartE2EDuration="3.179988529s" totalImagesPullingTime="2.005612911s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:10:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:10:16.176784418 +0000 UTC m=+629.758002996" watchObservedRunningTime="2026-08-04 18:10:16.179988529 +0000 UTC m=+629.761207107"
Aug 04 18:10:27.167815 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:10:27.167784    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-7b954ffc95-dwm6d"
Aug 04 18:12:08.416020 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:12:08.415985    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Aug 04 18:12:08.419632 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:12:08.419615    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Aug 04 18:12:08.423145 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:12:08.423120    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs\""
Aug 04 18:12:08.423145 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:12:08.423138    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"odh-kserve-custom-ca-bundle\""
Aug 04 18:12:08.424367 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:12:08.424348    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"default-dockercfg-vr8jv\""
Aug 04 18:12:08.424512 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:12:08.424377    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"kube-root-ca.crt\""
Aug 04 18:12:08.424512 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:12:08.424374    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisvc-model-fb-opt-125m-route-dc21cb-57cf7736-dockercfg-9trt8\""
Aug 04 18:12:08.424512 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:12:08.424434    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"openshift-service-ca.crt\""
Aug 04 18:12:08.424512 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:12:08.424436    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"seaweedfs-s3-creds\""
Aug 04 18:12:08.431780 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:12:08.431758    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Aug 04 18:12:08.594891 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:12:08.594848    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6019c4ff-1f97-481f-bb30-25faf41105a7-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"6019c4ff-1f97-481f-bb30-25faf41105a7\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Aug 04 18:12:08.594891 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:12:08.594894    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6019c4ff-1f97-481f-bb30-25faf41105a7-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"6019c4ff-1f97-481f-bb30-25faf41105a7\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Aug 04 18:12:08.595146 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:12:08.595060    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6019c4ff-1f97-481f-bb30-25faf41105a7-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"6019c4ff-1f97-481f-bb30-25faf41105a7\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Aug 04 18:12:08.595146 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:12:08.595119    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6019c4ff-1f97-481f-bb30-25faf41105a7-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"6019c4ff-1f97-481f-bb30-25faf41105a7\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Aug 04 18:12:08.595255 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:12:08.595170    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlw9h\" (UniqueName: \"kubernetes.io/projected/6019c4ff-1f97-481f-bb30-25faf41105a7-kube-api-access-nlw9h\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"6019c4ff-1f97-481f-bb30-25faf41105a7\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Aug 04 18:12:08.595255 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:12:08.595233    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6019c4ff-1f97-481f-bb30-25faf41105a7-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"6019c4ff-1f97-481f-bb30-25faf41105a7\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Aug 04 18:12:08.595353 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:12:08.595267    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6019c4ff-1f97-481f-bb30-25faf41105a7-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"6019c4ff-1f97-481f-bb30-25faf41105a7\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Aug 04 18:12:08.595353 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:12:08.595323    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6019c4ff-1f97-481f-bb30-25faf41105a7-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"6019c4ff-1f97-481f-bb30-25faf41105a7\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Aug 04 18:12:08.695897 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:12:08.695862    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6019c4ff-1f97-481f-bb30-25faf41105a7-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"6019c4ff-1f97-481f-bb30-25faf41105a7\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Aug 04 18:12:08.696056 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:12:08.695906    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6019c4ff-1f97-481f-bb30-25faf41105a7-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"6019c4ff-1f97-481f-bb30-25faf41105a7\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Aug 04 18:12:08.696056 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:12:08.696028    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlw9h\" (UniqueName: \"kubernetes.io/projected/6019c4ff-1f97-481f-bb30-25faf41105a7-kube-api-access-nlw9h\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"6019c4ff-1f97-481f-bb30-25faf41105a7\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Aug 04 18:12:08.696173 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:12:08.696111    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6019c4ff-1f97-481f-bb30-25faf41105a7-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"6019c4ff-1f97-481f-bb30-25faf41105a7\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Aug 04 18:12:08.696173 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:12:08.696150    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6019c4ff-1f97-481f-bb30-25faf41105a7-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"6019c4ff-1f97-481f-bb30-25faf41105a7\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Aug 04 18:12:08.696276 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:12:08.696180    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6019c4ff-1f97-481f-bb30-25faf41105a7-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"6019c4ff-1f97-481f-bb30-25faf41105a7\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Aug 04 18:12:08.696276 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:12:08.696228    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6019c4ff-1f97-481f-bb30-25faf41105a7-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"6019c4ff-1f97-481f-bb30-25faf41105a7\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Aug 04 18:12:08.696276 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:12:08.696261    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6019c4ff-1f97-481f-bb30-25faf41105a7-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"6019c4ff-1f97-481f-bb30-25faf41105a7\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Aug 04 18:12:08.696447 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:12:08.696266    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6019c4ff-1f97-481f-bb30-25faf41105a7-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"6019c4ff-1f97-481f-bb30-25faf41105a7\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Aug 04 18:12:08.696447 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:12:08.696411    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6019c4ff-1f97-481f-bb30-25faf41105a7-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"6019c4ff-1f97-481f-bb30-25faf41105a7\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Aug 04 18:12:08.696620 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:12:08.696595    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6019c4ff-1f97-481f-bb30-25faf41105a7-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"6019c4ff-1f97-481f-bb30-25faf41105a7\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Aug 04 18:12:08.696687 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:12:08.696659    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6019c4ff-1f97-481f-bb30-25faf41105a7-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"6019c4ff-1f97-481f-bb30-25faf41105a7\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Aug 04 18:12:08.696974 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:12:08.696947    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6019c4ff-1f97-481f-bb30-25faf41105a7-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"6019c4ff-1f97-481f-bb30-25faf41105a7\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Aug 04 18:12:08.698311 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:12:08.698289    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6019c4ff-1f97-481f-bb30-25faf41105a7-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"6019c4ff-1f97-481f-bb30-25faf41105a7\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Aug 04 18:12:08.698424 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:12:08.698406    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6019c4ff-1f97-481f-bb30-25faf41105a7-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"6019c4ff-1f97-481f-bb30-25faf41105a7\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Aug 04 18:12:08.716846 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:12:08.716825    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlw9h\" (UniqueName: \"kubernetes.io/projected/6019c4ff-1f97-481f-bb30-25faf41105a7-kube-api-access-nlw9h\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"6019c4ff-1f97-481f-bb30-25faf41105a7\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Aug 04 18:12:08.729380 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:12:08.729350    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Aug 04 18:12:08.874830 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:12:08.874803    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Aug 04 18:12:08.876273 ip-10-0-135-124 kubenswrapper[2583]: W0804 18:12:08.876249    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6019c4ff_1f97_481f_bb30_25faf41105a7.slice/crio-ca0df5fdcb7cc615b82207e716356160424902437ea152ee7af745cf27f16035 WatchSource:0}: Error finding container ca0df5fdcb7cc615b82207e716356160424902437ea152ee7af745cf27f16035: Status 404 returned error can't find the container with id ca0df5fdcb7cc615b82207e716356160424902437ea152ee7af745cf27f16035
Aug 04 18:12:08.878462 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:12:08.878446    2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Aug 04 18:12:09.755931 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:12:09.755894    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"6019c4ff-1f97-481f-bb30-25faf41105a7","Type":"ContainerStarted","Data":"ca0df5fdcb7cc615b82207e716356160424902437ea152ee7af745cf27f16035"}
Aug 04 18:12:12.778248 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:12:12.778202    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"6019c4ff-1f97-481f-bb30-25faf41105a7","Type":"ContainerStarted","Data":"bceb32f64c846297b1ad3664ab26b8395375ae891b807360829cef37a66a935f"}
Aug 04 18:12:14.790915 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:12:14.790876    2583 generic.go:356] "Generic (PLEG): container finished" podID="6019c4ff-1f97-481f-bb30-25faf41105a7" containerID="bceb32f64c846297b1ad3664ab26b8395375ae891b807360829cef37a66a935f" exitCode=0
Aug 04 18:12:14.790915 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:12:14.790916    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"6019c4ff-1f97-481f-bb30-25faf41105a7","Type":"ContainerDied","Data":"bceb32f64c846297b1ad3664ab26b8395375ae891b807360829cef37a66a935f"}
Aug 04 18:12:24.306928 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:12:24.306891    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-xr2wz"]
Aug 04 18:12:24.321756 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:12:24.321710    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-xr2wz"]
Aug 04 18:12:24.322028 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:12:24.321851    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-xr2wz"
Aug 04 18:12:24.325233 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:12:24.325180    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"kube-root-ca.crt\""
Aug 04 18:12:24.326865 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:12:24.326377    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"fc-smoke-test-epp-sa-dockercfg-8nw2j\""
Aug 04 18:12:24.326865 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:12:24.326413    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"openshift-service-ca.crt\""
Aug 04 18:12:24.326865 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:12:24.326694    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"fc-smoke-test-kserve-self-signed-certs\""
Aug 04 18:12:24.326865 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:12:24.326731    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"default-dockercfg-wbcwg\""
Aug 04 18:12:24.440019 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:12:24.439982    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmbtc\" (UniqueName: \"kubernetes.io/projected/8101344f-3f12-4673-8ffc-97625d244e3c-kube-api-access-vmbtc\") pod \"fc-smoke-test-kserve-router-scheduler-5cfd47986-xr2wz\" (UID: \"8101344f-3f12-4673-8ffc-97625d244e3c\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-xr2wz"
Aug 04 18:12:24.440177 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:12:24.440029    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8101344f-3f12-4673-8ffc-97625d244e3c-tls-certs\") pod \"fc-smoke-test-kserve-router-scheduler-5cfd47986-xr2wz\" (UID: \"8101344f-3f12-4673-8ffc-97625d244e3c\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-xr2wz"
Aug 04 18:12:24.541035 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:12:24.540993    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8101344f-3f12-4673-8ffc-97625d244e3c-tls-certs\") pod \"fc-smoke-test-kserve-router-scheduler-5cfd47986-xr2wz\" (UID: \"8101344f-3f12-4673-8ffc-97625d244e3c\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-xr2wz"
Aug 04 18:12:24.541220 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:12:24.541164    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmbtc\" (UniqueName: \"kubernetes.io/projected/8101344f-3f12-4673-8ffc-97625d244e3c-kube-api-access-vmbtc\") pod \"fc-smoke-test-kserve-router-scheduler-5cfd47986-xr2wz\" (UID: \"8101344f-3f12-4673-8ffc-97625d244e3c\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-xr2wz"
Aug 04 18:12:24.544199 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:12:24.544170    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8101344f-3f12-4673-8ffc-97625d244e3c-tls-certs\") pod \"fc-smoke-test-kserve-router-scheduler-5cfd47986-xr2wz\" (UID: \"8101344f-3f12-4673-8ffc-97625d244e3c\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-xr2wz"
Aug 04 18:12:24.559615 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:12:24.559555    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmbtc\" (UniqueName: \"kubernetes.io/projected/8101344f-3f12-4673-8ffc-97625d244e3c-kube-api-access-vmbtc\") pod \"fc-smoke-test-kserve-router-scheduler-5cfd47986-xr2wz\" (UID: \"8101344f-3f12-4673-8ffc-97625d244e3c\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-xr2wz"
Aug 04 18:12:24.635905 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:12:24.635871    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-xr2wz"
Aug 04 18:12:27.225017 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:12:27.224992    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-xr2wz"]
Aug 04 18:12:27.227421 ip-10-0-135-124 kubenswrapper[2583]: W0804 18:12:27.227382    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8101344f_3f12_4673_8ffc_97625d244e3c.slice/crio-1f65762f58ec637f1aa45fa74eea34136ebbf234db8540631995f7cc455446db WatchSource:0}: Error finding container 1f65762f58ec637f1aa45fa74eea34136ebbf234db8540631995f7cc455446db: Status 404 returned error can't find the container with id 1f65762f58ec637f1aa45fa74eea34136ebbf234db8540631995f7cc455446db
Aug 04 18:12:27.895025 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:12:27.894945    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-xr2wz" event={"ID":"8101344f-3f12-4673-8ffc-97625d244e3c","Type":"ContainerStarted","Data":"1f65762f58ec637f1aa45fa74eea34136ebbf234db8540631995f7cc455446db"}
Aug 04 18:12:42.992850 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:12:42.992808    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-xr2wz" event={"ID":"8101344f-3f12-4673-8ffc-97625d244e3c","Type":"ContainerStarted","Data":"16069d95d2aa4e00e30641af6875c910f08e2459b95c3c7056fb9707bd79b764"}
Aug 04 18:12:42.993309 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:12:42.993013    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-xr2wz"
Aug 04 18:12:42.995065 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:12:42.995041    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"6019c4ff-1f97-481f-bb30-25faf41105a7","Type":"ContainerStarted","Data":"c223ffa472129e8a00afa09a1033f063b937fd225bf514b8d7785d871e7ca2f8"}
Aug 04 18:12:43.010956 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:12:43.010911    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-xr2wz" podStartSLOduration=3.684332692 podStartE2EDuration="19.010895203s" totalImagesPullingTime="15.326562511s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:12:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:12:43.010103226 +0000 UTC m=+776.591321803" watchObservedRunningTime="2026-08-04 18:12:43.010895203 +0000 UTC m=+776.592113780"
Aug 04 18:12:43.035933 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:12:43.035888    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" podStartSLOduration=1.58634919 podStartE2EDuration="35.035873836s" totalImagesPullingTime="31.425594372s" totalInitContainerRuntime="2.023930274s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:12:08 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:12:43.032744987 +0000 UTC m=+776.613963564" watchObservedRunningTime="2026-08-04 18:12:43.035873836 +0000 UTC m=+776.617092412"
Aug 04 18:13:14.003089 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:13:14.003016    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-xr2wz"
Aug 04 18:13:20.649330 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:13:20.649290    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-xr2wz"]
Aug 04 18:13:24.002629 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:13:24.002585    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-xr2wz" podUID="8101344f-3f12-4673-8ffc-97625d244e3c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Aug 04 18:13:34.002642 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:13:34.002599    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-xr2wz" podUID="8101344f-3f12-4673-8ffc-97625d244e3c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Aug 04 18:13:35.671384 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:13:35.671311    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-xr2wz" podUID="8101344f-3f12-4673-8ffc-97625d244e3c" containerName="main" containerID="cri-o://16069d95d2aa4e00e30641af6875c910f08e2459b95c3c7056fb9707bd79b764" gracePeriod=45
Aug 04 18:13:35.917867 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:13:35.917821    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-xr2wz"
Aug 04 18:13:35.982222 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:13:35.982186    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8101344f-3f12-4673-8ffc-97625d244e3c-tls-certs\") pod \"8101344f-3f12-4673-8ffc-97625d244e3c\" (UID: \"8101344f-3f12-4673-8ffc-97625d244e3c\") "
Aug 04 18:13:35.982407 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:13:35.982343    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vmbtc\" (UniqueName: \"kubernetes.io/projected/8101344f-3f12-4673-8ffc-97625d244e3c-kube-api-access-vmbtc\") pod \"8101344f-3f12-4673-8ffc-97625d244e3c\" (UID: \"8101344f-3f12-4673-8ffc-97625d244e3c\") "
Aug 04 18:13:35.984486 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:13:35.984450    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8101344f-3f12-4673-8ffc-97625d244e3c-kube-api-access-vmbtc" (OuterVolumeSpecName: "kube-api-access-vmbtc") pod "8101344f-3f12-4673-8ffc-97625d244e3c" (UID: "8101344f-3f12-4673-8ffc-97625d244e3c"). InnerVolumeSpecName "kube-api-access-vmbtc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:13:35.984590 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:13:35.984512    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8101344f-3f12-4673-8ffc-97625d244e3c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8101344f-3f12-4673-8ffc-97625d244e3c" (UID: "8101344f-3f12-4673-8ffc-97625d244e3c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:13:36.084110 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:13:36.084070    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vmbtc\" (UniqueName: \"kubernetes.io/projected/8101344f-3f12-4673-8ffc-97625d244e3c-kube-api-access-vmbtc\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:13:36.084110 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:13:36.084114    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8101344f-3f12-4673-8ffc-97625d244e3c-tls-certs\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:13:36.295323 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:13:36.295243    2583 generic.go:356] "Generic (PLEG): container finished" podID="8101344f-3f12-4673-8ffc-97625d244e3c" containerID="16069d95d2aa4e00e30641af6875c910f08e2459b95c3c7056fb9707bd79b764" exitCode=0
Aug 04 18:13:36.295323 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:13:36.295302    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-xr2wz"
Aug 04 18:13:36.295543 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:13:36.295331    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-xr2wz" event={"ID":"8101344f-3f12-4673-8ffc-97625d244e3c","Type":"ContainerDied","Data":"16069d95d2aa4e00e30641af6875c910f08e2459b95c3c7056fb9707bd79b764"}
Aug 04 18:13:36.295543 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:13:36.295384    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-xr2wz" event={"ID":"8101344f-3f12-4673-8ffc-97625d244e3c","Type":"ContainerDied","Data":"1f65762f58ec637f1aa45fa74eea34136ebbf234db8540631995f7cc455446db"}
Aug 04 18:13:36.295543 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:13:36.295412    2583 scope.go:117] "RemoveContainer" containerID="16069d95d2aa4e00e30641af6875c910f08e2459b95c3c7056fb9707bd79b764"
Aug 04 18:13:36.304216 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:13:36.304197    2583 scope.go:117] "RemoveContainer" containerID="16069d95d2aa4e00e30641af6875c910f08e2459b95c3c7056fb9707bd79b764"
Aug 04 18:13:36.304461 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:13:36.304441    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16069d95d2aa4e00e30641af6875c910f08e2459b95c3c7056fb9707bd79b764\": container with ID starting with 16069d95d2aa4e00e30641af6875c910f08e2459b95c3c7056fb9707bd79b764 not found: ID does not exist" containerID="16069d95d2aa4e00e30641af6875c910f08e2459b95c3c7056fb9707bd79b764"
Aug 04 18:13:36.304547 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:13:36.304519    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16069d95d2aa4e00e30641af6875c910f08e2459b95c3c7056fb9707bd79b764"} err="failed to get container status \"16069d95d2aa4e00e30641af6875c910f08e2459b95c3c7056fb9707bd79b764\": rpc error: code = NotFound desc = could not find container \"16069d95d2aa4e00e30641af6875c910f08e2459b95c3c7056fb9707bd79b764\": container with ID starting with 16069d95d2aa4e00e30641af6875c910f08e2459b95c3c7056fb9707bd79b764 not found: ID does not exist"
Aug 04 18:13:36.323227 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:13:36.323204    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-xr2wz"]
Aug 04 18:13:36.328666 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:13:36.328642    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-5cfd47986-xr2wz"]
Aug 04 18:13:36.943281 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:13:36.943250    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8101344f-3f12-4673-8ffc-97625d244e3c" path="/var/lib/kubelet/pods/8101344f-3f12-4673-8ffc-97625d244e3c/volumes"
Aug 04 18:14:39.963127 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:14:39.963050    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Aug 04 18:14:46.923484 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:14:46.923354    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-wskfb_6157c2d3-2fd0-4544-88ad-434ac75f8c04/console-operator/1.log"
Aug 04 18:14:46.930353 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:14:46.925637    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-wskfb_6157c2d3-2fd0-4544-88ad-434ac75f8c04/console-operator/1.log"
Aug 04 18:14:54.992111 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:14:54.992047    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" podUID="6019c4ff-1f97-481f-bb30-25faf41105a7" containerName="main" containerID="cri-o://c223ffa472129e8a00afa09a1033f063b937fd225bf514b8d7785d871e7ca2f8" gracePeriod=45
Aug 04 18:14:55.741700 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:14:55.741672    2583 generic.go:356] "Generic (PLEG): container finished" podID="6019c4ff-1f97-481f-bb30-25faf41105a7" containerID="c223ffa472129e8a00afa09a1033f063b937fd225bf514b8d7785d871e7ca2f8" exitCode=0
Aug 04 18:14:55.741818 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:14:55.741733    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"6019c4ff-1f97-481f-bb30-25faf41105a7","Type":"ContainerDied","Data":"c223ffa472129e8a00afa09a1033f063b937fd225bf514b8d7785d871e7ca2f8"}
Aug 04 18:14:55.741818 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:14:55.741772    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"6019c4ff-1f97-481f-bb30-25faf41105a7","Type":"ContainerDied","Data":"ca0df5fdcb7cc615b82207e716356160424902437ea152ee7af745cf27f16035"}
Aug 04 18:14:55.741818 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:14:55.741794    2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ca0df5fdcb7cc615b82207e716356160424902437ea152ee7af745cf27f16035"
Aug 04 18:14:55.747556 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:14:55.747537    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Aug 04 18:14:55.774187 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:14:55.774113    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6019c4ff-1f97-481f-bb30-25faf41105a7-dshm\") pod \"6019c4ff-1f97-481f-bb30-25faf41105a7\" (UID: \"6019c4ff-1f97-481f-bb30-25faf41105a7\") "
Aug 04 18:14:55.774187 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:14:55.774156    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nlw9h\" (UniqueName: \"kubernetes.io/projected/6019c4ff-1f97-481f-bb30-25faf41105a7-kube-api-access-nlw9h\") pod \"6019c4ff-1f97-481f-bb30-25faf41105a7\" (UID: \"6019c4ff-1f97-481f-bb30-25faf41105a7\") "
Aug 04 18:14:55.774380 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:14:55.774191    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6019c4ff-1f97-481f-bb30-25faf41105a7-model-cache\") pod \"6019c4ff-1f97-481f-bb30-25faf41105a7\" (UID: \"6019c4ff-1f97-481f-bb30-25faf41105a7\") "
Aug 04 18:14:55.774380 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:14:55.774310    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6019c4ff-1f97-481f-bb30-25faf41105a7-home\") pod \"6019c4ff-1f97-481f-bb30-25faf41105a7\" (UID: \"6019c4ff-1f97-481f-bb30-25faf41105a7\") "
Aug 04 18:14:55.774380 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:14:55.774365    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6019c4ff-1f97-481f-bb30-25faf41105a7-tls-certs\") pod \"6019c4ff-1f97-481f-bb30-25faf41105a7\" (UID: \"6019c4ff-1f97-481f-bb30-25faf41105a7\") "
Aug 04 18:14:55.774574 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:14:55.774402    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6019c4ff-1f97-481f-bb30-25faf41105a7-tmp-dir\") pod \"6019c4ff-1f97-481f-bb30-25faf41105a7\" (UID: \"6019c4ff-1f97-481f-bb30-25faf41105a7\") "
Aug 04 18:14:55.774574 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:14:55.774429    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6019c4ff-1f97-481f-bb30-25faf41105a7-kserve-provision-location\") pod \"6019c4ff-1f97-481f-bb30-25faf41105a7\" (UID: \"6019c4ff-1f97-481f-bb30-25faf41105a7\") "
Aug 04 18:14:55.774574 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:14:55.774442    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6019c4ff-1f97-481f-bb30-25faf41105a7-model-cache" (OuterVolumeSpecName: "model-cache") pod "6019c4ff-1f97-481f-bb30-25faf41105a7" (UID: "6019c4ff-1f97-481f-bb30-25faf41105a7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:14:55.774574 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:14:55.774507    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6019c4ff-1f97-481f-bb30-25faf41105a7-cabundle-cert\") pod \"6019c4ff-1f97-481f-bb30-25faf41105a7\" (UID: \"6019c4ff-1f97-481f-bb30-25faf41105a7\") "
Aug 04 18:14:55.774945 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:14:55.774804    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6019c4ff-1f97-481f-bb30-25faf41105a7-model-cache\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:14:55.775234 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:14:55.775196    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6019c4ff-1f97-481f-bb30-25faf41105a7-home" (OuterVolumeSpecName: "home") pod "6019c4ff-1f97-481f-bb30-25faf41105a7" (UID: "6019c4ff-1f97-481f-bb30-25faf41105a7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:14:55.775328 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:14:55.775304    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6019c4ff-1f97-481f-bb30-25faf41105a7-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "6019c4ff-1f97-481f-bb30-25faf41105a7" (UID: "6019c4ff-1f97-481f-bb30-25faf41105a7"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:14:55.776888 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:14:55.776855    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6019c4ff-1f97-481f-bb30-25faf41105a7-dshm" (OuterVolumeSpecName: "dshm") pod "6019c4ff-1f97-481f-bb30-25faf41105a7" (UID: "6019c4ff-1f97-481f-bb30-25faf41105a7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:14:55.777604 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:14:55.777571    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6019c4ff-1f97-481f-bb30-25faf41105a7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6019c4ff-1f97-481f-bb30-25faf41105a7" (UID: "6019c4ff-1f97-481f-bb30-25faf41105a7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:14:55.777604 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:14:55.777594    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6019c4ff-1f97-481f-bb30-25faf41105a7-kube-api-access-nlw9h" (OuterVolumeSpecName: "kube-api-access-nlw9h") pod "6019c4ff-1f97-481f-bb30-25faf41105a7" (UID: "6019c4ff-1f97-481f-bb30-25faf41105a7"). InnerVolumeSpecName "kube-api-access-nlw9h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:14:55.795838 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:14:55.795811    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6019c4ff-1f97-481f-bb30-25faf41105a7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6019c4ff-1f97-481f-bb30-25faf41105a7" (UID: "6019c4ff-1f97-481f-bb30-25faf41105a7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:14:55.797458 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:14:55.797437    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6019c4ff-1f97-481f-bb30-25faf41105a7-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6019c4ff-1f97-481f-bb30-25faf41105a7" (UID: "6019c4ff-1f97-481f-bb30-25faf41105a7"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:14:55.875743 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:14:55.875719    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6019c4ff-1f97-481f-bb30-25faf41105a7-dshm\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:14:55.875743 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:14:55.875742    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nlw9h\" (UniqueName: \"kubernetes.io/projected/6019c4ff-1f97-481f-bb30-25faf41105a7-kube-api-access-nlw9h\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:14:55.875880 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:14:55.875752    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6019c4ff-1f97-481f-bb30-25faf41105a7-home\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:14:55.875880 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:14:55.875761    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6019c4ff-1f97-481f-bb30-25faf41105a7-tls-certs\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:14:55.875880 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:14:55.875769    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6019c4ff-1f97-481f-bb30-25faf41105a7-tmp-dir\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:14:55.875880 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:14:55.875778    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6019c4ff-1f97-481f-bb30-25faf41105a7-kserve-provision-location\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:14:55.875880 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:14:55.875786    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6019c4ff-1f97-481f-bb30-25faf41105a7-cabundle-cert\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:14:56.747573 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:14:56.747541    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Aug 04 18:14:56.772024 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:14:56.771987    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Aug 04 18:14:56.774996 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:14:56.774969    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Aug 04 18:14:56.942523 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:14:56.942493    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6019c4ff-1f97-481f-bb30-25faf41105a7" path="/var/lib/kubelet/pods/6019c4ff-1f97-481f-bb30-25faf41105a7/volumes"
Aug 04 18:15:10.598806 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:15:10.598772    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-2rpgr"]
Aug 04 18:15:10.600026 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:15:10.599998    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6019c4ff-1f97-481f-bb30-25faf41105a7" containerName="main"
Aug 04 18:15:10.600183 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:15:10.600171    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="6019c4ff-1f97-481f-bb30-25faf41105a7" containerName="main"
Aug 04 18:15:10.600285 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:15:10.600275    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8101344f-3f12-4673-8ffc-97625d244e3c" containerName="main"
Aug 04 18:15:10.600355 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:15:10.600347    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="8101344f-3f12-4673-8ffc-97625d244e3c" containerName="main"
Aug 04 18:15:10.600488 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:15:10.600461    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6019c4ff-1f97-481f-bb30-25faf41105a7" containerName="storage-initializer"
Aug 04 18:15:10.600577 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:15:10.600567    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="6019c4ff-1f97-481f-bb30-25faf41105a7" containerName="storage-initializer"
Aug 04 18:15:10.600843 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:15:10.600831    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="8101344f-3f12-4673-8ffc-97625d244e3c" containerName="main"
Aug 04 18:15:10.600927 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:15:10.600918    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="6019c4ff-1f97-481f-bb30-25faf41105a7" containerName="main"
Aug 04 18:15:10.606928 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:15:10.606883    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-2rpgr"
Aug 04 18:15:10.609825 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:15:10.609760    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-2rpgr"]
Aug 04 18:15:10.631276 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:15:10.631254    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"kube-root-ca.crt\""
Aug 04 18:15:10.631768 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:15:10.631748    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"default-dockercfg-sc5dn\""
Aug 04 18:15:10.631857 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:15:10.631748    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"openshift-service-ca.crt\""
Aug 04 18:15:10.631857 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:15:10.631805    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"gw-sec0c69dceeb48768325d1a53a749e65786-kserve-self-signed-certs\""
Aug 04 18:15:10.692943 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:15:10.692917    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/056cf9ae-0ebd-422d-90cd-002fd0b7f3bf-home\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-2rpgr\" (UID: \"056cf9ae-0ebd-422d-90cd-002fd0b7f3bf\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-2rpgr"
Aug 04 18:15:10.693056 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:15:10.693005    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/056cf9ae-0ebd-422d-90cd-002fd0b7f3bf-tmp-dir\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-2rpgr\" (UID: \"056cf9ae-0ebd-422d-90cd-002fd0b7f3bf\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-2rpgr"
Aug 04 18:15:10.693056 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:15:10.693049    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/056cf9ae-0ebd-422d-90cd-002fd0b7f3bf-dshm\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-2rpgr\" (UID: \"056cf9ae-0ebd-422d-90cd-002fd0b7f3bf\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-2rpgr"
Aug 04 18:15:10.693138 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:15:10.693076    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xcgbv\" (UniqueName: \"kubernetes.io/projected/056cf9ae-0ebd-422d-90cd-002fd0b7f3bf-kube-api-access-xcgbv\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-2rpgr\" (UID: \"056cf9ae-0ebd-422d-90cd-002fd0b7f3bf\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-2rpgr"
Aug 04 18:15:10.693138 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:15:10.693129    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/056cf9ae-0ebd-422d-90cd-002fd0b7f3bf-tls-certs\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-2rpgr\" (UID: \"056cf9ae-0ebd-422d-90cd-002fd0b7f3bf\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-2rpgr"
Aug 04 18:15:10.693260 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:15:10.693160    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/056cf9ae-0ebd-422d-90cd-002fd0b7f3bf-model-cache\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-2rpgr\" (UID: \"056cf9ae-0ebd-422d-90cd-002fd0b7f3bf\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-2rpgr"
Aug 04 18:15:10.794099 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:15:10.794072    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/056cf9ae-0ebd-422d-90cd-002fd0b7f3bf-tmp-dir\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-2rpgr\" (UID: \"056cf9ae-0ebd-422d-90cd-002fd0b7f3bf\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-2rpgr"
Aug 04 18:15:10.794213 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:15:10.794108    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/056cf9ae-0ebd-422d-90cd-002fd0b7f3bf-dshm\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-2rpgr\" (UID: \"056cf9ae-0ebd-422d-90cd-002fd0b7f3bf\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-2rpgr"
Aug 04 18:15:10.794213 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:15:10.794127    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xcgbv\" (UniqueName: \"kubernetes.io/projected/056cf9ae-0ebd-422d-90cd-002fd0b7f3bf-kube-api-access-xcgbv\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-2rpgr\" (UID: \"056cf9ae-0ebd-422d-90cd-002fd0b7f3bf\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-2rpgr"
Aug 04 18:15:10.794213 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:15:10.794157    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/056cf9ae-0ebd-422d-90cd-002fd0b7f3bf-tls-certs\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-2rpgr\" (UID: \"056cf9ae-0ebd-422d-90cd-002fd0b7f3bf\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-2rpgr"
Aug 04 18:15:10.794357 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:15:10.794212    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/056cf9ae-0ebd-422d-90cd-002fd0b7f3bf-model-cache\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-2rpgr\" (UID: \"056cf9ae-0ebd-422d-90cd-002fd0b7f3bf\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-2rpgr"
Aug 04 18:15:10.794357 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:15:10.794288    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/056cf9ae-0ebd-422d-90cd-002fd0b7f3bf-home\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-2rpgr\" (UID: \"056cf9ae-0ebd-422d-90cd-002fd0b7f3bf\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-2rpgr"
Aug 04 18:15:10.794553 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:15:10.794534    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/056cf9ae-0ebd-422d-90cd-002fd0b7f3bf-tmp-dir\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-2rpgr\" (UID: \"056cf9ae-0ebd-422d-90cd-002fd0b7f3bf\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-2rpgr"
Aug 04 18:15:10.794631 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:15:10.794607    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/056cf9ae-0ebd-422d-90cd-002fd0b7f3bf-model-cache\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-2rpgr\" (UID: \"056cf9ae-0ebd-422d-90cd-002fd0b7f3bf\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-2rpgr"
Aug 04 18:15:10.794688 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:15:10.794670    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/056cf9ae-0ebd-422d-90cd-002fd0b7f3bf-home\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-2rpgr\" (UID: \"056cf9ae-0ebd-422d-90cd-002fd0b7f3bf\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-2rpgr"
Aug 04 18:15:10.796448 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:15:10.796417    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/056cf9ae-0ebd-422d-90cd-002fd0b7f3bf-dshm\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-2rpgr\" (UID: \"056cf9ae-0ebd-422d-90cd-002fd0b7f3bf\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-2rpgr"
Aug 04 18:15:10.796735 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:15:10.796716    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/056cf9ae-0ebd-422d-90cd-002fd0b7f3bf-tls-certs\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-2rpgr\" (UID: \"056cf9ae-0ebd-422d-90cd-002fd0b7f3bf\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-2rpgr"
Aug 04 18:15:10.802136 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:15:10.802115    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xcgbv\" (UniqueName: \"kubernetes.io/projected/056cf9ae-0ebd-422d-90cd-002fd0b7f3bf-kube-api-access-xcgbv\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-2rpgr\" (UID: \"056cf9ae-0ebd-422d-90cd-002fd0b7f3bf\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-2rpgr"
Aug 04 18:15:10.918221 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:15:10.918151    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-2rpgr"
Aug 04 18:15:11.058435 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:15:11.058401    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-2rpgr"]
Aug 04 18:15:11.060525 ip-10-0-135-124 kubenswrapper[2583]: W0804 18:15:11.060499    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod056cf9ae_0ebd_422d_90cd_002fd0b7f3bf.slice/crio-f1efff5f7c68d20850f0376e2304a59272deca4c5e991e68bfbaae195f9783a3 WatchSource:0}: Error finding container f1efff5f7c68d20850f0376e2304a59272deca4c5e991e68bfbaae195f9783a3: Status 404 returned error can't find the container with id f1efff5f7c68d20850f0376e2304a59272deca4c5e991e68bfbaae195f9783a3
Aug 04 18:15:11.833188 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:15:11.833150    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-2rpgr" event={"ID":"056cf9ae-0ebd-422d-90cd-002fd0b7f3bf","Type":"ContainerStarted","Data":"f1efff5f7c68d20850f0376e2304a59272deca4c5e991e68bfbaae195f9783a3"}
Aug 04 18:15:12.839943 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:15:12.839911    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-2rpgr" event={"ID":"056cf9ae-0ebd-422d-90cd-002fd0b7f3bf","Type":"ContainerStarted","Data":"41301d56ba961d04b2198e616cd3966d6c1a68326340aa2550ea0e551f3356ba"}
Aug 04 18:15:12.859996 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:15:12.859950    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-2rpgr" podStartSLOduration=1.207089129 podStartE2EDuration="2.859934034s" totalImagesPullingTime="1.652844905s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:15:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:15:12.859566857 +0000 UTC m=+926.440785436" watchObservedRunningTime="2026-08-04 18:15:12.859934034 +0000 UTC m=+926.441152611"
Aug 04 18:15:12.918707 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:15:12.918646    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-2rpgr"
Aug 04 18:15:17.341762 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:15:17.341728    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-2rpgr"]
Aug 04 18:15:17.351255 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:15:17.351220    2583 log.go:32] "ExecSync cmd from runtime service failed" err=<
Aug 04 18:15:17.351255 ip-10-0-135-124 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Aug 04 18:15:17.351255 ip-10-0-135-124 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Aug 04 18:15:17.351255 ip-10-0-135-124 kubenswrapper[2583]:  > containerID="41301d56ba961d04b2198e616cd3966d6c1a68326340aa2550ea0e551f3356ba" cmd=["/bin/sleep","15"]
Aug 04 18:15:17.351481 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:15:17.351275    2583 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-2rpgr" message="" err=<
Aug 04 18:15:17.351481 ip-10-0-135-124 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Aug 04 18:15:17.351481 ip-10-0-135-124 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Aug 04 18:15:17.351481 ip-10-0-135-124 kubenswrapper[2583]:  >
Aug 04 18:15:17.351481 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:15:17.351288    2583 kuberuntime_container.go:755] "PreStop hook failed" err=<
Aug 04 18:15:17.351481 ip-10-0-135-124 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Aug 04 18:15:17.351481 ip-10-0-135-124 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Aug 04 18:15:17.351481 ip-10-0-135-124 kubenswrapper[2583]:  > pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-2rpgr" podUID="056cf9ae-0ebd-422d-90cd-002fd0b7f3bf" containerName="main" containerID="cri-o://41301d56ba961d04b2198e616cd3966d6c1a68326340aa2550ea0e551f3356ba"
Aug 04 18:15:17.351481 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:15:17.351324    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-2rpgr" podUID="056cf9ae-0ebd-422d-90cd-002fd0b7f3bf" containerName="main" containerID="cri-o://41301d56ba961d04b2198e616cd3966d6c1a68326340aa2550ea0e551f3356ba" gracePeriod=60
Aug 04 18:15:17.705109 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:15:17.705086    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-2rpgr"
Aug 04 18:15:17.757812 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:15:17.757785    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/056cf9ae-0ebd-422d-90cd-002fd0b7f3bf-dshm\") pod \"056cf9ae-0ebd-422d-90cd-002fd0b7f3bf\" (UID: \"056cf9ae-0ebd-422d-90cd-002fd0b7f3bf\") "
Aug 04 18:15:17.757925 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:15:17.757841    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/056cf9ae-0ebd-422d-90cd-002fd0b7f3bf-tmp-dir\") pod \"056cf9ae-0ebd-422d-90cd-002fd0b7f3bf\" (UID: \"056cf9ae-0ebd-422d-90cd-002fd0b7f3bf\") "
Aug 04 18:15:17.757925 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:15:17.757876    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/056cf9ae-0ebd-422d-90cd-002fd0b7f3bf-model-cache\") pod \"056cf9ae-0ebd-422d-90cd-002fd0b7f3bf\" (UID: \"056cf9ae-0ebd-422d-90cd-002fd0b7f3bf\") "
Aug 04 18:15:17.757996 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:15:17.757916    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/056cf9ae-0ebd-422d-90cd-002fd0b7f3bf-tls-certs\") pod \"056cf9ae-0ebd-422d-90cd-002fd0b7f3bf\" (UID: \"056cf9ae-0ebd-422d-90cd-002fd0b7f3bf\") "
Aug 04 18:15:17.757996 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:15:17.757983    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/056cf9ae-0ebd-422d-90cd-002fd0b7f3bf-home\") pod \"056cf9ae-0ebd-422d-90cd-002fd0b7f3bf\" (UID: \"056cf9ae-0ebd-422d-90cd-002fd0b7f3bf\") "
Aug 04 18:15:17.758068 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:15:17.758015    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xcgbv\" (UniqueName: \"kubernetes.io/projected/056cf9ae-0ebd-422d-90cd-002fd0b7f3bf-kube-api-access-xcgbv\") pod \"056cf9ae-0ebd-422d-90cd-002fd0b7f3bf\" (UID: \"056cf9ae-0ebd-422d-90cd-002fd0b7f3bf\") "
Aug 04 18:15:17.758177 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:15:17.758148    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/056cf9ae-0ebd-422d-90cd-002fd0b7f3bf-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "056cf9ae-0ebd-422d-90cd-002fd0b7f3bf" (UID: "056cf9ae-0ebd-422d-90cd-002fd0b7f3bf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:15:17.758249 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:15:17.758158    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/056cf9ae-0ebd-422d-90cd-002fd0b7f3bf-model-cache" (OuterVolumeSpecName: "model-cache") pod "056cf9ae-0ebd-422d-90cd-002fd0b7f3bf" (UID: "056cf9ae-0ebd-422d-90cd-002fd0b7f3bf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:15:17.758309 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:15:17.758252    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/056cf9ae-0ebd-422d-90cd-002fd0b7f3bf-home" (OuterVolumeSpecName: "home") pod "056cf9ae-0ebd-422d-90cd-002fd0b7f3bf" (UID: "056cf9ae-0ebd-422d-90cd-002fd0b7f3bf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:15:17.758309 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:15:17.758295    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/056cf9ae-0ebd-422d-90cd-002fd0b7f3bf-tmp-dir\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:15:17.758410 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:15:17.758316    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/056cf9ae-0ebd-422d-90cd-002fd0b7f3bf-model-cache\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:15:17.759923 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:15:17.759895    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/056cf9ae-0ebd-422d-90cd-002fd0b7f3bf-dshm" (OuterVolumeSpecName: "dshm") pod "056cf9ae-0ebd-422d-90cd-002fd0b7f3bf" (UID: "056cf9ae-0ebd-422d-90cd-002fd0b7f3bf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:15:17.760072 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:15:17.760055    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/056cf9ae-0ebd-422d-90cd-002fd0b7f3bf-kube-api-access-xcgbv" (OuterVolumeSpecName: "kube-api-access-xcgbv") pod "056cf9ae-0ebd-422d-90cd-002fd0b7f3bf" (UID: "056cf9ae-0ebd-422d-90cd-002fd0b7f3bf"). InnerVolumeSpecName "kube-api-access-xcgbv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:15:17.760168 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:15:17.760146    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/056cf9ae-0ebd-422d-90cd-002fd0b7f3bf-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "056cf9ae-0ebd-422d-90cd-002fd0b7f3bf" (UID: "056cf9ae-0ebd-422d-90cd-002fd0b7f3bf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:15:17.858883 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:15:17.858860    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/056cf9ae-0ebd-422d-90cd-002fd0b7f3bf-home\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:15:17.858883 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:15:17.858881    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xcgbv\" (UniqueName: \"kubernetes.io/projected/056cf9ae-0ebd-422d-90cd-002fd0b7f3bf-kube-api-access-xcgbv\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:15:17.859017 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:15:17.858892    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/056cf9ae-0ebd-422d-90cd-002fd0b7f3bf-dshm\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:15:17.859017 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:15:17.858900    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/056cf9ae-0ebd-422d-90cd-002fd0b7f3bf-tls-certs\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:15:17.866891 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:15:17.866867    2583 generic.go:356] "Generic (PLEG): container finished" podID="056cf9ae-0ebd-422d-90cd-002fd0b7f3bf" containerID="41301d56ba961d04b2198e616cd3966d6c1a68326340aa2550ea0e551f3356ba" exitCode=0
Aug 04 18:15:17.867000 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:15:17.866909    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-2rpgr"
Aug 04 18:15:17.867000 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:15:17.866960    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-2rpgr" event={"ID":"056cf9ae-0ebd-422d-90cd-002fd0b7f3bf","Type":"ContainerDied","Data":"41301d56ba961d04b2198e616cd3966d6c1a68326340aa2550ea0e551f3356ba"}
Aug 04 18:15:17.867073 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:15:17.867017    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-2rpgr" event={"ID":"056cf9ae-0ebd-422d-90cd-002fd0b7f3bf","Type":"ContainerDied","Data":"f1efff5f7c68d20850f0376e2304a59272deca4c5e991e68bfbaae195f9783a3"}
Aug 04 18:15:17.867073 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:15:17.867044    2583 scope.go:117] "RemoveContainer" containerID="41301d56ba961d04b2198e616cd3966d6c1a68326340aa2550ea0e551f3356ba"
Aug 04 18:15:17.876294 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:15:17.876279    2583 scope.go:117] "RemoveContainer" containerID="41301d56ba961d04b2198e616cd3966d6c1a68326340aa2550ea0e551f3356ba"
Aug 04 18:15:17.876560 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:15:17.876543    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41301d56ba961d04b2198e616cd3966d6c1a68326340aa2550ea0e551f3356ba\": container with ID starting with 41301d56ba961d04b2198e616cd3966d6c1a68326340aa2550ea0e551f3356ba not found: ID does not exist" containerID="41301d56ba961d04b2198e616cd3966d6c1a68326340aa2550ea0e551f3356ba"
Aug 04 18:15:17.876618 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:15:17.876571    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41301d56ba961d04b2198e616cd3966d6c1a68326340aa2550ea0e551f3356ba"} err="failed to get container status \"41301d56ba961d04b2198e616cd3966d6c1a68326340aa2550ea0e551f3356ba\": rpc error: code = NotFound desc = could not find container \"41301d56ba961d04b2198e616cd3966d6c1a68326340aa2550ea0e551f3356ba\": container with ID starting with 41301d56ba961d04b2198e616cd3966d6c1a68326340aa2550ea0e551f3356ba not found: ID does not exist"
Aug 04 18:15:17.888949 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:15:17.888927    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-2rpgr"]
Aug 04 18:15:17.890921 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:15:17.890902    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-2rpgr"]
Aug 04 18:15:18.941427 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:15:18.941394    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="056cf9ae-0ebd-422d-90cd-002fd0b7f3bf" path="/var/lib/kubelet/pods/056cf9ae-0ebd-422d-90cd-002fd0b7f3bf/volumes"
Aug 04 18:15:24.785314 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:15:24.785279    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-2kjnb"]
Aug 04 18:15:24.785801 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:15:24.785679    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="056cf9ae-0ebd-422d-90cd-002fd0b7f3bf" containerName="main"
Aug 04 18:15:24.785801 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:15:24.785691    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="056cf9ae-0ebd-422d-90cd-002fd0b7f3bf" containerName="main"
Aug 04 18:15:24.785801 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:15:24.785755    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="056cf9ae-0ebd-422d-90cd-002fd0b7f3bf" containerName="main"
Aug 04 18:15:24.790100 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:15:24.790080    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-2kjnb"
Aug 04 18:15:24.800239 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:15:24.800214    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-2kjnb"]
Aug 04 18:15:24.811113 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:15:24.811086    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"openshift-service-ca.crt\""
Aug 04 18:15:24.811113 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:15:24.811107    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"kube-root-ca.crt\""
Aug 04 18:15:24.811282 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:15:24.811263    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"default-dockercfg-lp2pq\""
Aug 04 18:15:24.811330 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:15:24.811263    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"scheduler-inline-config-test-kserve-self-signed-certs\""
Aug 04 18:15:24.918424 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:15:24.918397    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a9f5540-1b91-4904-bcab-590badbcdb82-tls-certs\") pod \"scheduler-inline-config-test-kserve-8655bc757-2kjnb\" (UID: \"5a9f5540-1b91-4904-bcab-590badbcdb82\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-2kjnb"
Aug 04 18:15:24.918600 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:15:24.918433    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a9f5540-1b91-4904-bcab-590badbcdb82-home\") pod \"scheduler-inline-config-test-kserve-8655bc757-2kjnb\" (UID: \"5a9f5540-1b91-4904-bcab-590badbcdb82\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-2kjnb"
Aug 04 18:15:24.918600 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:15:24.918514    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a9f5540-1b91-4904-bcab-590badbcdb82-dshm\") pod \"scheduler-inline-config-test-kserve-8655bc757-2kjnb\" (UID: \"5a9f5540-1b91-4904-bcab-590badbcdb82\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-2kjnb"
Aug 04 18:15:24.918600 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:15:24.918575    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnj2w\" (UniqueName: \"kubernetes.io/projected/5a9f5540-1b91-4904-bcab-590badbcdb82-kube-api-access-nnj2w\") pod \"scheduler-inline-config-test-kserve-8655bc757-2kjnb\" (UID: \"5a9f5540-1b91-4904-bcab-590badbcdb82\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-2kjnb"
Aug 04 18:15:24.918780 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:15:24.918675    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a9f5540-1b91-4904-bcab-590badbcdb82-model-cache\") pod \"scheduler-inline-config-test-kserve-8655bc757-2kjnb\" (UID: \"5a9f5540-1b91-4904-bcab-590badbcdb82\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-2kjnb"
Aug 04 18:15:24.918780 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:15:24.918723    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a9f5540-1b91-4904-bcab-590badbcdb82-tmp-dir\") pod \"scheduler-inline-config-test-kserve-8655bc757-2kjnb\" (UID: \"5a9f5540-1b91-4904-bcab-590badbcdb82\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-2kjnb"
Aug 04 18:15:25.019672 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:15:25.019643    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a9f5540-1b91-4904-bcab-590badbcdb82-model-cache\") pod \"scheduler-inline-config-test-kserve-8655bc757-2kjnb\" (UID: \"5a9f5540-1b91-4904-bcab-590badbcdb82\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-2kjnb"
Aug 04 18:15:25.019801 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:15:25.019680    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a9f5540-1b91-4904-bcab-590badbcdb82-tmp-dir\") pod \"scheduler-inline-config-test-kserve-8655bc757-2kjnb\" (UID: \"5a9f5540-1b91-4904-bcab-590badbcdb82\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-2kjnb"
Aug 04 18:15:25.019801 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:15:25.019701    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a9f5540-1b91-4904-bcab-590badbcdb82-tls-certs\") pod \"scheduler-inline-config-test-kserve-8655bc757-2kjnb\" (UID: \"5a9f5540-1b91-4904-bcab-590badbcdb82\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-2kjnb"
Aug 04 18:15:25.019801 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:15:25.019719    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a9f5540-1b91-4904-bcab-590badbcdb82-home\") pod \"scheduler-inline-config-test-kserve-8655bc757-2kjnb\" (UID: \"5a9f5540-1b91-4904-bcab-590badbcdb82\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-2kjnb"
Aug 04 18:15:25.019801 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:15:25.019746    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a9f5540-1b91-4904-bcab-590badbcdb82-dshm\") pod \"scheduler-inline-config-test-kserve-8655bc757-2kjnb\" (UID: \"5a9f5540-1b91-4904-bcab-590badbcdb82\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-2kjnb"
Aug 04 18:15:25.019801 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:15:25.019783    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnj2w\" (UniqueName: \"kubernetes.io/projected/5a9f5540-1b91-4904-bcab-590badbcdb82-kube-api-access-nnj2w\") pod \"scheduler-inline-config-test-kserve-8655bc757-2kjnb\" (UID: \"5a9f5540-1b91-4904-bcab-590badbcdb82\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-2kjnb"
Aug 04 18:15:25.020084 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:15:25.020058    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a9f5540-1b91-4904-bcab-590badbcdb82-model-cache\") pod \"scheduler-inline-config-test-kserve-8655bc757-2kjnb\" (UID: \"5a9f5540-1b91-4904-bcab-590badbcdb82\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-2kjnb"
Aug 04 18:15:25.020153 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:15:25.020097    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a9f5540-1b91-4904-bcab-590badbcdb82-tmp-dir\") pod \"scheduler-inline-config-test-kserve-8655bc757-2kjnb\" (UID: \"5a9f5540-1b91-4904-bcab-590badbcdb82\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-2kjnb"
Aug 04 18:15:25.020277 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:15:25.020253    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a9f5540-1b91-4904-bcab-590badbcdb82-home\") pod \"scheduler-inline-config-test-kserve-8655bc757-2kjnb\" (UID: \"5a9f5540-1b91-4904-bcab-590badbcdb82\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-2kjnb"
Aug 04 18:15:25.021973 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:15:25.021950    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a9f5540-1b91-4904-bcab-590badbcdb82-dshm\") pod \"scheduler-inline-config-test-kserve-8655bc757-2kjnb\" (UID: \"5a9f5540-1b91-4904-bcab-590badbcdb82\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-2kjnb"
Aug 04 18:15:25.022143 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:15:25.022127    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a9f5540-1b91-4904-bcab-590badbcdb82-tls-certs\") pod \"scheduler-inline-config-test-kserve-8655bc757-2kjnb\" (UID: \"5a9f5540-1b91-4904-bcab-590badbcdb82\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-2kjnb"
Aug 04 18:15:25.029482 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:15:25.029445    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnj2w\" (UniqueName: \"kubernetes.io/projected/5a9f5540-1b91-4904-bcab-590badbcdb82-kube-api-access-nnj2w\") pod \"scheduler-inline-config-test-kserve-8655bc757-2kjnb\" (UID: \"5a9f5540-1b91-4904-bcab-590badbcdb82\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-2kjnb"
Aug 04 18:15:25.100439 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:15:25.100387    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-2kjnb"
Aug 04 18:15:25.246220 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:15:25.246190    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-2kjnb"]
Aug 04 18:15:25.247588 ip-10-0-135-124 kubenswrapper[2583]: W0804 18:15:25.247562    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a9f5540_1b91_4904_bcab_590badbcdb82.slice/crio-c917885ba97315724873147703089d8350296ef97e964d6f3bfc90648c6123ab WatchSource:0}: Error finding container c917885ba97315724873147703089d8350296ef97e964d6f3bfc90648c6123ab: Status 404 returned error can't find the container with id c917885ba97315724873147703089d8350296ef97e964d6f3bfc90648c6123ab
Aug 04 18:15:25.913627 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:15:25.913585    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-2kjnb" event={"ID":"5a9f5540-1b91-4904-bcab-590badbcdb82","Type":"ContainerStarted","Data":"be0754ad08a3da829be58346a5463d694453b1062b9b241bb7872dd1ec786452"}
Aug 04 18:15:25.914001 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:15:25.913651    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-2kjnb" event={"ID":"5a9f5540-1b91-4904-bcab-590badbcdb82","Type":"ContainerStarted","Data":"c917885ba97315724873147703089d8350296ef97e964d6f3bfc90648c6123ab"}
Aug 04 18:15:25.932876 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:15:25.932830    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-2kjnb" podStartSLOduration=1.932814709 podStartE2EDuration="1.932814709s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:15:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:15:25.931211851 +0000 UTC m=+939.512430431" watchObservedRunningTime="2026-08-04 18:15:25.932814709 +0000 UTC m=+939.514033287"
Aug 04 18:15:26.100580 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:15:26.100549    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-2kjnb"
Aug 04 18:15:35.100913 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:15:35.100875    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-2kjnb"
Aug 04 18:15:35.113522 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:15:35.113490    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-2kjnb"
Aug 04 18:15:35.982389 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:15:35.982360    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-2kjnb"
Aug 04 18:16:13.004072 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:16:13.003971    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-2kjnb"]
Aug 04 18:16:13.013909 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:16:13.013868    2583 log.go:32] "ExecSync cmd from runtime service failed" err=<
Aug 04 18:16:13.013909 ip-10-0-135-124 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Aug 04 18:16:13.013909 ip-10-0-135-124 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Aug 04 18:16:13.013909 ip-10-0-135-124 kubenswrapper[2583]:  > containerID="be0754ad08a3da829be58346a5463d694453b1062b9b241bb7872dd1ec786452" cmd=["/bin/sleep","15"]
Aug 04 18:16:13.014213 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:16:13.013933    2583 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-2kjnb" message="" err=<
Aug 04 18:16:13.014213 ip-10-0-135-124 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Aug 04 18:16:13.014213 ip-10-0-135-124 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Aug 04 18:16:13.014213 ip-10-0-135-124 kubenswrapper[2583]:  >
Aug 04 18:16:13.014213 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:16:13.013950    2583 kuberuntime_container.go:755] "PreStop hook failed" err=<
Aug 04 18:16:13.014213 ip-10-0-135-124 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Aug 04 18:16:13.014213 ip-10-0-135-124 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Aug 04 18:16:13.014213 ip-10-0-135-124 kubenswrapper[2583]:  > pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-2kjnb" podUID="5a9f5540-1b91-4904-bcab-590badbcdb82" containerName="main" containerID="cri-o://be0754ad08a3da829be58346a5463d694453b1062b9b241bb7872dd1ec786452"
Aug 04 18:16:13.014213 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:16:13.014006    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-2kjnb" podUID="5a9f5540-1b91-4904-bcab-590badbcdb82" containerName="main" containerID="cri-o://be0754ad08a3da829be58346a5463d694453b1062b9b241bb7872dd1ec786452" gracePeriod=60
Aug 04 18:16:13.365541 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:16:13.365444    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-2kjnb"
Aug 04 18:16:13.465967 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:16:13.465930    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a9f5540-1b91-4904-bcab-590badbcdb82-tmp-dir\") pod \"5a9f5540-1b91-4904-bcab-590badbcdb82\" (UID: \"5a9f5540-1b91-4904-bcab-590badbcdb82\") "
Aug 04 18:16:13.465967 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:16:13.465971    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a9f5540-1b91-4904-bcab-590badbcdb82-tls-certs\") pod \"5a9f5540-1b91-4904-bcab-590badbcdb82\" (UID: \"5a9f5540-1b91-4904-bcab-590badbcdb82\") "
Aug 04 18:16:13.466170 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:16:13.466016    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a9f5540-1b91-4904-bcab-590badbcdb82-dshm\") pod \"5a9f5540-1b91-4904-bcab-590badbcdb82\" (UID: \"5a9f5540-1b91-4904-bcab-590badbcdb82\") "
Aug 04 18:16:13.466170 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:16:13.466063    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nnj2w\" (UniqueName: \"kubernetes.io/projected/5a9f5540-1b91-4904-bcab-590badbcdb82-kube-api-access-nnj2w\") pod \"5a9f5540-1b91-4904-bcab-590badbcdb82\" (UID: \"5a9f5540-1b91-4904-bcab-590badbcdb82\") "
Aug 04 18:16:13.466170 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:16:13.466115    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a9f5540-1b91-4904-bcab-590badbcdb82-model-cache\") pod \"5a9f5540-1b91-4904-bcab-590badbcdb82\" (UID: \"5a9f5540-1b91-4904-bcab-590badbcdb82\") "
Aug 04 18:16:13.466170 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:16:13.466147    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a9f5540-1b91-4904-bcab-590badbcdb82-home\") pod \"5a9f5540-1b91-4904-bcab-590badbcdb82\" (UID: \"5a9f5540-1b91-4904-bcab-590badbcdb82\") "
Aug 04 18:16:13.466385 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:16:13.466222    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a9f5540-1b91-4904-bcab-590badbcdb82-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5a9f5540-1b91-4904-bcab-590badbcdb82" (UID: "5a9f5540-1b91-4904-bcab-590badbcdb82"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:16:13.466452 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:16:13.466400    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a9f5540-1b91-4904-bcab-590badbcdb82-home" (OuterVolumeSpecName: "home") pod "5a9f5540-1b91-4904-bcab-590badbcdb82" (UID: "5a9f5540-1b91-4904-bcab-590badbcdb82"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:16:13.466452 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:16:13.466417    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a9f5540-1b91-4904-bcab-590badbcdb82-model-cache" (OuterVolumeSpecName: "model-cache") pod "5a9f5540-1b91-4904-bcab-590badbcdb82" (UID: "5a9f5540-1b91-4904-bcab-590badbcdb82"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:16:13.466452 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:16:13.466438    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a9f5540-1b91-4904-bcab-590badbcdb82-tmp-dir\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:16:13.468358 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:16:13.468332    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a9f5540-1b91-4904-bcab-590badbcdb82-dshm" (OuterVolumeSpecName: "dshm") pod "5a9f5540-1b91-4904-bcab-590badbcdb82" (UID: "5a9f5540-1b91-4904-bcab-590badbcdb82"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:16:13.468462 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:16:13.468380    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a9f5540-1b91-4904-bcab-590badbcdb82-kube-api-access-nnj2w" (OuterVolumeSpecName: "kube-api-access-nnj2w") pod "5a9f5540-1b91-4904-bcab-590badbcdb82" (UID: "5a9f5540-1b91-4904-bcab-590badbcdb82"). InnerVolumeSpecName "kube-api-access-nnj2w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:16:13.468462 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:16:13.468399    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a9f5540-1b91-4904-bcab-590badbcdb82-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5a9f5540-1b91-4904-bcab-590badbcdb82" (UID: "5a9f5540-1b91-4904-bcab-590badbcdb82"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:16:13.567415 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:16:13.567334    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a9f5540-1b91-4904-bcab-590badbcdb82-dshm\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:16:13.567415 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:16:13.567366    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nnj2w\" (UniqueName: \"kubernetes.io/projected/5a9f5540-1b91-4904-bcab-590badbcdb82-kube-api-access-nnj2w\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:16:13.567415 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:16:13.567379    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a9f5540-1b91-4904-bcab-590badbcdb82-model-cache\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:16:13.567415 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:16:13.567388    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a9f5540-1b91-4904-bcab-590badbcdb82-home\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:16:13.567415 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:16:13.567396    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a9f5540-1b91-4904-bcab-590badbcdb82-tls-certs\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:16:14.180311 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:16:14.180278    2583 generic.go:356] "Generic (PLEG): container finished" podID="5a9f5540-1b91-4904-bcab-590badbcdb82" containerID="be0754ad08a3da829be58346a5463d694453b1062b9b241bb7872dd1ec786452" exitCode=0
Aug 04 18:16:14.180806 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:16:14.180327    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-2kjnb"
Aug 04 18:16:14.180806 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:16:14.180361    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-2kjnb" event={"ID":"5a9f5540-1b91-4904-bcab-590badbcdb82","Type":"ContainerDied","Data":"be0754ad08a3da829be58346a5463d694453b1062b9b241bb7872dd1ec786452"}
Aug 04 18:16:14.180806 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:16:14.180413    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-2kjnb" event={"ID":"5a9f5540-1b91-4904-bcab-590badbcdb82","Type":"ContainerDied","Data":"c917885ba97315724873147703089d8350296ef97e964d6f3bfc90648c6123ab"}
Aug 04 18:16:14.180806 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:16:14.180434    2583 scope.go:117] "RemoveContainer" containerID="be0754ad08a3da829be58346a5463d694453b1062b9b241bb7872dd1ec786452"
Aug 04 18:16:14.190337 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:16:14.190317    2583 scope.go:117] "RemoveContainer" containerID="be0754ad08a3da829be58346a5463d694453b1062b9b241bb7872dd1ec786452"
Aug 04 18:16:14.190640 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:16:14.190621    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be0754ad08a3da829be58346a5463d694453b1062b9b241bb7872dd1ec786452\": container with ID starting with be0754ad08a3da829be58346a5463d694453b1062b9b241bb7872dd1ec786452 not found: ID does not exist" containerID="be0754ad08a3da829be58346a5463d694453b1062b9b241bb7872dd1ec786452"
Aug 04 18:16:14.190697 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:16:14.190649    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be0754ad08a3da829be58346a5463d694453b1062b9b241bb7872dd1ec786452"} err="failed to get container status \"be0754ad08a3da829be58346a5463d694453b1062b9b241bb7872dd1ec786452\": rpc error: code = NotFound desc = could not find container \"be0754ad08a3da829be58346a5463d694453b1062b9b241bb7872dd1ec786452\": container with ID starting with be0754ad08a3da829be58346a5463d694453b1062b9b241bb7872dd1ec786452 not found: ID does not exist"
Aug 04 18:16:14.204297 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:16:14.204268    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-2kjnb"]
Aug 04 18:16:14.208215 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:16:14.208193    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-2kjnb"]
Aug 04 18:16:14.941877 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:16:14.941846    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5a9f5540-1b91-4904-bcab-590badbcdb82" path="/var/lib/kubelet/pods/5a9f5540-1b91-4904-bcab-590badbcdb82/volumes"
Aug 04 18:16:43.297730 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:16:43.297685    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fxg5r4"]
Aug 04 18:16:43.298113 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:16:43.298036    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a9f5540-1b91-4904-bcab-590badbcdb82" containerName="main"
Aug 04 18:16:43.298113 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:16:43.298048    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a9f5540-1b91-4904-bcab-590badbcdb82" containerName="main"
Aug 04 18:16:43.298192 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:16:43.298117    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a9f5540-1b91-4904-bcab-590badbcdb82" containerName="main"
Aug 04 18:16:43.301262 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:16:43.301244    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fxg5r4"
Aug 04 18:16:43.305527 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:16:43.305502    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"kube-root-ca.crt\""
Aug 04 18:16:43.305680 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:16:43.305505    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"llmisv431d7e30f13554296364403a5b6365f5-kserve-self-signed-certs\""
Aug 04 18:16:43.305680 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:16:43.305508    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"default-dockercfg-j9vvv\""
Aug 04 18:16:43.305680 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:16:43.305505    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"openshift-service-ca.crt\""
Aug 04 18:16:43.312923 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:16:43.312901    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fxg5r4"]
Aug 04 18:16:43.397846 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:16:43.397814    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/83c6288b-0c4c-4ebf-86da-e08d31a3514a-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fxg5r4\" (UID: \"83c6288b-0c4c-4ebf-86da-e08d31a3514a\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fxg5r4"
Aug 04 18:16:43.398018 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:16:43.397852    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/83c6288b-0c4c-4ebf-86da-e08d31a3514a-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fxg5r4\" (UID: \"83c6288b-0c4c-4ebf-86da-e08d31a3514a\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fxg5r4"
Aug 04 18:16:43.398018 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:16:43.397871    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/83c6288b-0c4c-4ebf-86da-e08d31a3514a-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fxg5r4\" (UID: \"83c6288b-0c4c-4ebf-86da-e08d31a3514a\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fxg5r4"
Aug 04 18:16:43.398018 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:16:43.397905    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/83c6288b-0c4c-4ebf-86da-e08d31a3514a-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fxg5r4\" (UID: \"83c6288b-0c4c-4ebf-86da-e08d31a3514a\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fxg5r4"
Aug 04 18:16:43.398018 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:16:43.397957    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjw4r\" (UniqueName: \"kubernetes.io/projected/83c6288b-0c4c-4ebf-86da-e08d31a3514a-kube-api-access-vjw4r\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fxg5r4\" (UID: \"83c6288b-0c4c-4ebf-86da-e08d31a3514a\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fxg5r4"
Aug 04 18:16:43.398018 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:16:43.398009    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/83c6288b-0c4c-4ebf-86da-e08d31a3514a-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fxg5r4\" (UID: \"83c6288b-0c4c-4ebf-86da-e08d31a3514a\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fxg5r4"
Aug 04 18:16:43.498810 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:16:43.498777    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/83c6288b-0c4c-4ebf-86da-e08d31a3514a-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fxg5r4\" (UID: \"83c6288b-0c4c-4ebf-86da-e08d31a3514a\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fxg5r4"
Aug 04 18:16:43.498976 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:16:43.498830    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/83c6288b-0c4c-4ebf-86da-e08d31a3514a-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fxg5r4\" (UID: \"83c6288b-0c4c-4ebf-86da-e08d31a3514a\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fxg5r4"
Aug 04 18:16:43.498976 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:16:43.498864    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/83c6288b-0c4c-4ebf-86da-e08d31a3514a-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fxg5r4\" (UID: \"83c6288b-0c4c-4ebf-86da-e08d31a3514a\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fxg5r4"
Aug 04 18:16:43.498976 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:16:43.498888    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/83c6288b-0c4c-4ebf-86da-e08d31a3514a-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fxg5r4\" (UID: \"83c6288b-0c4c-4ebf-86da-e08d31a3514a\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fxg5r4"
Aug 04 18:16:43.498976 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:16:43.498921    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/83c6288b-0c4c-4ebf-86da-e08d31a3514a-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fxg5r4\" (UID: \"83c6288b-0c4c-4ebf-86da-e08d31a3514a\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fxg5r4"
Aug 04 18:16:43.498976 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:16:43.498956    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjw4r\" (UniqueName: \"kubernetes.io/projected/83c6288b-0c4c-4ebf-86da-e08d31a3514a-kube-api-access-vjw4r\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fxg5r4\" (UID: \"83c6288b-0c4c-4ebf-86da-e08d31a3514a\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fxg5r4"
Aug 04 18:16:43.499239 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:16:43.499219    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/83c6288b-0c4c-4ebf-86da-e08d31a3514a-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fxg5r4\" (UID: \"83c6288b-0c4c-4ebf-86da-e08d31a3514a\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fxg5r4"
Aug 04 18:16:43.499304 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:16:43.499278    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/83c6288b-0c4c-4ebf-86da-e08d31a3514a-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fxg5r4\" (UID: \"83c6288b-0c4c-4ebf-86da-e08d31a3514a\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fxg5r4"
Aug 04 18:16:43.499382 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:16:43.499286    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/83c6288b-0c4c-4ebf-86da-e08d31a3514a-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fxg5r4\" (UID: \"83c6288b-0c4c-4ebf-86da-e08d31a3514a\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fxg5r4"
Aug 04 18:16:43.501243 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:16:43.501217    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/83c6288b-0c4c-4ebf-86da-e08d31a3514a-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fxg5r4\" (UID: \"83c6288b-0c4c-4ebf-86da-e08d31a3514a\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fxg5r4"
Aug 04 18:16:43.501500 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:16:43.501484    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/83c6288b-0c4c-4ebf-86da-e08d31a3514a-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fxg5r4\" (UID: \"83c6288b-0c4c-4ebf-86da-e08d31a3514a\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fxg5r4"
Aug 04 18:16:43.508235 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:16:43.508209    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjw4r\" (UniqueName: \"kubernetes.io/projected/83c6288b-0c4c-4ebf-86da-e08d31a3514a-kube-api-access-vjw4r\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fxg5r4\" (UID: \"83c6288b-0c4c-4ebf-86da-e08d31a3514a\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fxg5r4"
Aug 04 18:16:43.612659 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:16:43.612587    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fxg5r4"
Aug 04 18:16:43.748505 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:16:43.747980    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fxg5r4"]
Aug 04 18:16:43.750544 ip-10-0-135-124 kubenswrapper[2583]: W0804 18:16:43.750513    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83c6288b_0c4c_4ebf_86da_e08d31a3514a.slice/crio-ba23af37ec02cdea6882b8dd43fcad2351e5840cafb1a8b22c75ff8c7bd85711 WatchSource:0}: Error finding container ba23af37ec02cdea6882b8dd43fcad2351e5840cafb1a8b22c75ff8c7bd85711: Status 404 returned error can't find the container with id ba23af37ec02cdea6882b8dd43fcad2351e5840cafb1a8b22c75ff8c7bd85711
Aug 04 18:16:44.345138 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:16:44.345097    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fxg5r4" event={"ID":"83c6288b-0c4c-4ebf-86da-e08d31a3514a","Type":"ContainerStarted","Data":"25c6fe935dbae663bb3ba722a4fb0859b6566426773c844dafe94cc403709c31"}
Aug 04 18:16:44.345587 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:16:44.345165    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fxg5r4" event={"ID":"83c6288b-0c4c-4ebf-86da-e08d31a3514a","Type":"ContainerStarted","Data":"ba23af37ec02cdea6882b8dd43fcad2351e5840cafb1a8b22c75ff8c7bd85711"}
Aug 04 18:16:44.370003 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:16:44.369942    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fxg5r4" podStartSLOduration=1.369922143 podStartE2EDuration="1.369922143s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:16:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:16:44.365083457 +0000 UTC m=+1017.946302037" watchObservedRunningTime="2026-08-04 18:16:44.369922143 +0000 UTC m=+1017.951140723"
Aug 04 18:16:44.613135 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:16:44.613051    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fxg5r4"
Aug 04 18:16:53.613424 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:16:53.613386    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fxg5r4"
Aug 04 18:16:53.625594 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:16:53.625570    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fxg5r4"
Aug 04 18:16:54.415523 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:16:54.415489    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fxg5r4"
Aug 04 18:17:16.862712 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:17:16.862677    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fxg5r4"]
Aug 04 18:17:16.871665 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:17:16.871622    2583 log.go:32] "ExecSync cmd from runtime service failed" err=<
Aug 04 18:17:16.871665 ip-10-0-135-124 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Aug 04 18:17:16.871665 ip-10-0-135-124 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Aug 04 18:17:16.871665 ip-10-0-135-124 kubenswrapper[2583]:  > containerID="25c6fe935dbae663bb3ba722a4fb0859b6566426773c844dafe94cc403709c31" cmd=["/bin/sleep","15"]
Aug 04 18:17:16.871946 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:17:16.871681    2583 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fxg5r4" message="" err=<
Aug 04 18:17:16.871946 ip-10-0-135-124 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Aug 04 18:17:16.871946 ip-10-0-135-124 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Aug 04 18:17:16.871946 ip-10-0-135-124 kubenswrapper[2583]:  >
Aug 04 18:17:16.871946 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:17:16.871698    2583 kuberuntime_container.go:755] "PreStop hook failed" err=<
Aug 04 18:17:16.871946 ip-10-0-135-124 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Aug 04 18:17:16.871946 ip-10-0-135-124 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Aug 04 18:17:16.871946 ip-10-0-135-124 kubenswrapper[2583]:  > pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fxg5r4" podUID="83c6288b-0c4c-4ebf-86da-e08d31a3514a" containerName="main" containerID="cri-o://25c6fe935dbae663bb3ba722a4fb0859b6566426773c844dafe94cc403709c31"
Aug 04 18:17:16.871946 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:17:16.871752    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fxg5r4" podUID="83c6288b-0c4c-4ebf-86da-e08d31a3514a" containerName="main" containerID="cri-o://25c6fe935dbae663bb3ba722a4fb0859b6566426773c844dafe94cc403709c31" gracePeriod=60
Aug 04 18:17:17.217813 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:17:17.217789    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fxg5r4"
Aug 04 18:17:17.385071 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:17:17.384996    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/83c6288b-0c4c-4ebf-86da-e08d31a3514a-dshm\") pod \"83c6288b-0c4c-4ebf-86da-e08d31a3514a\" (UID: \"83c6288b-0c4c-4ebf-86da-e08d31a3514a\") "
Aug 04 18:17:17.385071 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:17:17.385034    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/83c6288b-0c4c-4ebf-86da-e08d31a3514a-home\") pod \"83c6288b-0c4c-4ebf-86da-e08d31a3514a\" (UID: \"83c6288b-0c4c-4ebf-86da-e08d31a3514a\") "
Aug 04 18:17:17.385071 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:17:17.385055    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/83c6288b-0c4c-4ebf-86da-e08d31a3514a-tmp-dir\") pod \"83c6288b-0c4c-4ebf-86da-e08d31a3514a\" (UID: \"83c6288b-0c4c-4ebf-86da-e08d31a3514a\") "
Aug 04 18:17:17.385335 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:17:17.385097    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/83c6288b-0c4c-4ebf-86da-e08d31a3514a-model-cache\") pod \"83c6288b-0c4c-4ebf-86da-e08d31a3514a\" (UID: \"83c6288b-0c4c-4ebf-86da-e08d31a3514a\") "
Aug 04 18:17:17.385335 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:17:17.385126    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vjw4r\" (UniqueName: \"kubernetes.io/projected/83c6288b-0c4c-4ebf-86da-e08d31a3514a-kube-api-access-vjw4r\") pod \"83c6288b-0c4c-4ebf-86da-e08d31a3514a\" (UID: \"83c6288b-0c4c-4ebf-86da-e08d31a3514a\") "
Aug 04 18:17:17.385335 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:17:17.385155    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/83c6288b-0c4c-4ebf-86da-e08d31a3514a-tls-certs\") pod \"83c6288b-0c4c-4ebf-86da-e08d31a3514a\" (UID: \"83c6288b-0c4c-4ebf-86da-e08d31a3514a\") "
Aug 04 18:17:17.385335 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:17:17.385298    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83c6288b-0c4c-4ebf-86da-e08d31a3514a-home" (OuterVolumeSpecName: "home") pod "83c6288b-0c4c-4ebf-86da-e08d31a3514a" (UID: "83c6288b-0c4c-4ebf-86da-e08d31a3514a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:17:17.385586 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:17:17.385334    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83c6288b-0c4c-4ebf-86da-e08d31a3514a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "83c6288b-0c4c-4ebf-86da-e08d31a3514a" (UID: "83c6288b-0c4c-4ebf-86da-e08d31a3514a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:17:17.385586 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:17:17.385382    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83c6288b-0c4c-4ebf-86da-e08d31a3514a-model-cache" (OuterVolumeSpecName: "model-cache") pod "83c6288b-0c4c-4ebf-86da-e08d31a3514a" (UID: "83c6288b-0c4c-4ebf-86da-e08d31a3514a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:17:17.387322 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:17:17.387291    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83c6288b-0c4c-4ebf-86da-e08d31a3514a-dshm" (OuterVolumeSpecName: "dshm") pod "83c6288b-0c4c-4ebf-86da-e08d31a3514a" (UID: "83c6288b-0c4c-4ebf-86da-e08d31a3514a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:17:17.387436 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:17:17.387343    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83c6288b-0c4c-4ebf-86da-e08d31a3514a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "83c6288b-0c4c-4ebf-86da-e08d31a3514a" (UID: "83c6288b-0c4c-4ebf-86da-e08d31a3514a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:17:17.387436 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:17:17.387383    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83c6288b-0c4c-4ebf-86da-e08d31a3514a-kube-api-access-vjw4r" (OuterVolumeSpecName: "kube-api-access-vjw4r") pod "83c6288b-0c4c-4ebf-86da-e08d31a3514a" (UID: "83c6288b-0c4c-4ebf-86da-e08d31a3514a"). InnerVolumeSpecName "kube-api-access-vjw4r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:17:17.485762 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:17:17.485726    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/83c6288b-0c4c-4ebf-86da-e08d31a3514a-model-cache\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:17:17.485762 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:17:17.485757    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vjw4r\" (UniqueName: \"kubernetes.io/projected/83c6288b-0c4c-4ebf-86da-e08d31a3514a-kube-api-access-vjw4r\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:17:17.485958 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:17:17.485772    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/83c6288b-0c4c-4ebf-86da-e08d31a3514a-tls-certs\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:17:17.485958 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:17:17.485785    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/83c6288b-0c4c-4ebf-86da-e08d31a3514a-dshm\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:17:17.485958 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:17:17.485796    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/83c6288b-0c4c-4ebf-86da-e08d31a3514a-home\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:17:17.485958 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:17:17.485805    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/83c6288b-0c4c-4ebf-86da-e08d31a3514a-tmp-dir\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:17:17.527909 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:17:17.527874    2583 generic.go:356] "Generic (PLEG): container finished" podID="83c6288b-0c4c-4ebf-86da-e08d31a3514a" containerID="25c6fe935dbae663bb3ba722a4fb0859b6566426773c844dafe94cc403709c31" exitCode=0
Aug 04 18:17:17.528059 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:17:17.527916    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fxg5r4" event={"ID":"83c6288b-0c4c-4ebf-86da-e08d31a3514a","Type":"ContainerDied","Data":"25c6fe935dbae663bb3ba722a4fb0859b6566426773c844dafe94cc403709c31"}
Aug 04 18:17:17.528059 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:17:17.527923    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fxg5r4"
Aug 04 18:17:17.528059 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:17:17.527970    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fxg5r4" event={"ID":"83c6288b-0c4c-4ebf-86da-e08d31a3514a","Type":"ContainerDied","Data":"ba23af37ec02cdea6882b8dd43fcad2351e5840cafb1a8b22c75ff8c7bd85711"}
Aug 04 18:17:17.528059 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:17:17.528000    2583 scope.go:117] "RemoveContainer" containerID="25c6fe935dbae663bb3ba722a4fb0859b6566426773c844dafe94cc403709c31"
Aug 04 18:17:17.538049 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:17:17.538032    2583 scope.go:117] "RemoveContainer" containerID="25c6fe935dbae663bb3ba722a4fb0859b6566426773c844dafe94cc403709c31"
Aug 04 18:17:17.538294 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:17:17.538275    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25c6fe935dbae663bb3ba722a4fb0859b6566426773c844dafe94cc403709c31\": container with ID starting with 25c6fe935dbae663bb3ba722a4fb0859b6566426773c844dafe94cc403709c31 not found: ID does not exist" containerID="25c6fe935dbae663bb3ba722a4fb0859b6566426773c844dafe94cc403709c31"
Aug 04 18:17:17.538360 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:17:17.538300    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25c6fe935dbae663bb3ba722a4fb0859b6566426773c844dafe94cc403709c31"} err="failed to get container status \"25c6fe935dbae663bb3ba722a4fb0859b6566426773c844dafe94cc403709c31\": rpc error: code = NotFound desc = could not find container \"25c6fe935dbae663bb3ba722a4fb0859b6566426773c844dafe94cc403709c31\": container with ID starting with 25c6fe935dbae663bb3ba722a4fb0859b6566426773c844dafe94cc403709c31 not found: ID does not exist"
Aug 04 18:17:17.553236 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:17:17.553212    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fxg5r4"]
Aug 04 18:17:17.570678 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:17:17.570653    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fxg5r4"]
Aug 04 18:17:18.941752 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:17:18.941721    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="83c6288b-0c4c-4ebf-86da-e08d31a3514a" path="/var/lib/kubelet/pods/83c6288b-0c4c-4ebf-86da-e08d31a3514a/volumes"
Aug 04 18:17:43.407155 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:17:43.407117    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-kd4kg"]
Aug 04 18:17:43.407616 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:17:43.407509    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83c6288b-0c4c-4ebf-86da-e08d31a3514a" containerName="main"
Aug 04 18:17:43.407616 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:17:43.407521    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="83c6288b-0c4c-4ebf-86da-e08d31a3514a" containerName="main"
Aug 04 18:17:43.407616 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:17:43.407580    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="83c6288b-0c4c-4ebf-86da-e08d31a3514a" containerName="main"
Aug 04 18:17:43.411938 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:17:43.411918    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-kd4kg"
Aug 04 18:17:43.416380 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:17:43.416361    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"kube-root-ca.crt\""
Aug 04 18:17:43.416514 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:17:43.416363    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"default-dockercfg-nlj6g\""
Aug 04 18:17:43.416514 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:17:43.416399    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"openshift-service-ca.crt\""
Aug 04 18:17:43.416514 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:17:43.416404    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"scheduler-configmap-ref-test-kserve-self-signed-certs\""
Aug 04 18:17:43.421352 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:17:43.421274    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-kd4kg"]
Aug 04 18:17:43.494068 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:17:43.494034    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f1be45d-006a-4441-98e7-53d6c28c8f57-tmp-dir\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-kd4kg\" (UID: \"4f1be45d-006a-4441-98e7-53d6c28c8f57\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-kd4kg"
Aug 04 18:17:43.494255 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:17:43.494086    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f1be45d-006a-4441-98e7-53d6c28c8f57-home\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-kd4kg\" (UID: \"4f1be45d-006a-4441-98e7-53d6c28c8f57\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-kd4kg"
Aug 04 18:17:43.494255 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:17:43.494116    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f1be45d-006a-4441-98e7-53d6c28c8f57-model-cache\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-kd4kg\" (UID: \"4f1be45d-006a-4441-98e7-53d6c28c8f57\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-kd4kg"
Aug 04 18:17:43.494255 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:17:43.494136    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f1be45d-006a-4441-98e7-53d6c28c8f57-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-kd4kg\" (UID: \"4f1be45d-006a-4441-98e7-53d6c28c8f57\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-kd4kg"
Aug 04 18:17:43.494255 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:17:43.494194    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f1be45d-006a-4441-98e7-53d6c28c8f57-dshm\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-kd4kg\" (UID: \"4f1be45d-006a-4441-98e7-53d6c28c8f57\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-kd4kg"
Aug 04 18:17:43.494255 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:17:43.494232    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gsm5m\" (UniqueName: \"kubernetes.io/projected/4f1be45d-006a-4441-98e7-53d6c28c8f57-kube-api-access-gsm5m\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-kd4kg\" (UID: \"4f1be45d-006a-4441-98e7-53d6c28c8f57\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-kd4kg"
Aug 04 18:17:43.595689 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:17:43.595646    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f1be45d-006a-4441-98e7-53d6c28c8f57-home\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-kd4kg\" (UID: \"4f1be45d-006a-4441-98e7-53d6c28c8f57\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-kd4kg"
Aug 04 18:17:43.595689 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:17:43.595701    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f1be45d-006a-4441-98e7-53d6c28c8f57-model-cache\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-kd4kg\" (UID: \"4f1be45d-006a-4441-98e7-53d6c28c8f57\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-kd4kg"
Aug 04 18:17:43.595954 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:17:43.595729    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f1be45d-006a-4441-98e7-53d6c28c8f57-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-kd4kg\" (UID: \"4f1be45d-006a-4441-98e7-53d6c28c8f57\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-kd4kg"
Aug 04 18:17:43.595954 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:17:43.595765    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f1be45d-006a-4441-98e7-53d6c28c8f57-dshm\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-kd4kg\" (UID: \"4f1be45d-006a-4441-98e7-53d6c28c8f57\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-kd4kg"
Aug 04 18:17:43.595954 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:17:43.595805    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gsm5m\" (UniqueName: \"kubernetes.io/projected/4f1be45d-006a-4441-98e7-53d6c28c8f57-kube-api-access-gsm5m\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-kd4kg\" (UID: \"4f1be45d-006a-4441-98e7-53d6c28c8f57\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-kd4kg"
Aug 04 18:17:43.595954 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:17:43.595921    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f1be45d-006a-4441-98e7-53d6c28c8f57-tmp-dir\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-kd4kg\" (UID: \"4f1be45d-006a-4441-98e7-53d6c28c8f57\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-kd4kg"
Aug 04 18:17:43.596180 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:17:43.596125    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f1be45d-006a-4441-98e7-53d6c28c8f57-model-cache\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-kd4kg\" (UID: \"4f1be45d-006a-4441-98e7-53d6c28c8f57\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-kd4kg"
Aug 04 18:17:43.596180 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:17:43.596168    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f1be45d-006a-4441-98e7-53d6c28c8f57-home\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-kd4kg\" (UID: \"4f1be45d-006a-4441-98e7-53d6c28c8f57\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-kd4kg"
Aug 04 18:17:43.596288 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:17:43.596198    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f1be45d-006a-4441-98e7-53d6c28c8f57-tmp-dir\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-kd4kg\" (UID: \"4f1be45d-006a-4441-98e7-53d6c28c8f57\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-kd4kg"
Aug 04 18:17:43.598281 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:17:43.598255    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f1be45d-006a-4441-98e7-53d6c28c8f57-dshm\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-kd4kg\" (UID: \"4f1be45d-006a-4441-98e7-53d6c28c8f57\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-kd4kg"
Aug 04 18:17:43.598699 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:17:43.598678    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f1be45d-006a-4441-98e7-53d6c28c8f57-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-kd4kg\" (UID: \"4f1be45d-006a-4441-98e7-53d6c28c8f57\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-kd4kg"
Aug 04 18:17:43.606954 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:17:43.606929    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gsm5m\" (UniqueName: \"kubernetes.io/projected/4f1be45d-006a-4441-98e7-53d6c28c8f57-kube-api-access-gsm5m\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-kd4kg\" (UID: \"4f1be45d-006a-4441-98e7-53d6c28c8f57\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-kd4kg"
Aug 04 18:17:43.722831 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:17:43.722794    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-kd4kg"
Aug 04 18:17:43.864336 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:17:43.864301    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-kd4kg"]
Aug 04 18:17:43.869219 ip-10-0-135-124 kubenswrapper[2583]: W0804 18:17:43.869191    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f1be45d_006a_4441_98e7_53d6c28c8f57.slice/crio-e628e92f53dde8adc4b46fba324ad28e67e29136f44cd3c4f0312e8308fdff1a WatchSource:0}: Error finding container e628e92f53dde8adc4b46fba324ad28e67e29136f44cd3c4f0312e8308fdff1a: Status 404 returned error can't find the container with id e628e92f53dde8adc4b46fba324ad28e67e29136f44cd3c4f0312e8308fdff1a
Aug 04 18:17:44.676111 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:17:44.676076    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-kd4kg" event={"ID":"4f1be45d-006a-4441-98e7-53d6c28c8f57","Type":"ContainerStarted","Data":"b14b2b32e77cc3a59dab029d54afa3d58065fc8e4190d2445b44c9e29275dc62"}
Aug 04 18:17:44.676537 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:17:44.676128    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-kd4kg" event={"ID":"4f1be45d-006a-4441-98e7-53d6c28c8f57","Type":"ContainerStarted","Data":"e628e92f53dde8adc4b46fba324ad28e67e29136f44cd3c4f0312e8308fdff1a"}
Aug 04 18:17:44.694434 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:17:44.694392    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-kd4kg" podStartSLOduration=1.69437754 podStartE2EDuration="1.69437754s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:17:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:17:44.693173742 +0000 UTC m=+1078.274392321" watchObservedRunningTime="2026-08-04 18:17:44.69437754 +0000 UTC m=+1078.275596118"
Aug 04 18:17:44.723567 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:17:44.723541    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-kd4kg"
Aug 04 18:17:53.723821 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:17:53.723786    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-kd4kg"
Aug 04 18:17:53.738132 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:17:53.738105    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-kd4kg"
Aug 04 18:17:53.749098 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:17:53.749078    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-kd4kg"
Aug 04 18:18:17.022085 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:18:17.022052    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-kd4kg"]
Aug 04 18:18:17.031162 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:18:17.031122    2583 log.go:32] "ExecSync cmd from runtime service failed" err=<
Aug 04 18:18:17.031162 ip-10-0-135-124 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Aug 04 18:18:17.031162 ip-10-0-135-124 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Aug 04 18:18:17.031162 ip-10-0-135-124 kubenswrapper[2583]:  > containerID="b14b2b32e77cc3a59dab029d54afa3d58065fc8e4190d2445b44c9e29275dc62" cmd=["/bin/sleep","15"]
Aug 04 18:18:17.031450 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:18:17.031179    2583 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-kd4kg" message="" err=<
Aug 04 18:18:17.031450 ip-10-0-135-124 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Aug 04 18:18:17.031450 ip-10-0-135-124 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Aug 04 18:18:17.031450 ip-10-0-135-124 kubenswrapper[2583]:  >
Aug 04 18:18:17.031450 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:18:17.031196    2583 kuberuntime_container.go:755] "PreStop hook failed" err=<
Aug 04 18:18:17.031450 ip-10-0-135-124 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Aug 04 18:18:17.031450 ip-10-0-135-124 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Aug 04 18:18:17.031450 ip-10-0-135-124 kubenswrapper[2583]:  > pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-kd4kg" podUID="4f1be45d-006a-4441-98e7-53d6c28c8f57" containerName="main" containerID="cri-o://b14b2b32e77cc3a59dab029d54afa3d58065fc8e4190d2445b44c9e29275dc62"
Aug 04 18:18:17.031450 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:18:17.031250    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-kd4kg" podUID="4f1be45d-006a-4441-98e7-53d6c28c8f57" containerName="main" containerID="cri-o://b14b2b32e77cc3a59dab029d54afa3d58065fc8e4190d2445b44c9e29275dc62" gracePeriod=60
Aug 04 18:18:17.373285 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:18:17.373259    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-kd4kg"
Aug 04 18:18:17.493064 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:18:17.493033    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f1be45d-006a-4441-98e7-53d6c28c8f57-tls-certs\") pod \"4f1be45d-006a-4441-98e7-53d6c28c8f57\" (UID: \"4f1be45d-006a-4441-98e7-53d6c28c8f57\") "
Aug 04 18:18:17.493064 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:18:17.493067    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f1be45d-006a-4441-98e7-53d6c28c8f57-dshm\") pod \"4f1be45d-006a-4441-98e7-53d6c28c8f57\" (UID: \"4f1be45d-006a-4441-98e7-53d6c28c8f57\") "
Aug 04 18:18:17.493286 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:18:17.493106    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f1be45d-006a-4441-98e7-53d6c28c8f57-model-cache\") pod \"4f1be45d-006a-4441-98e7-53d6c28c8f57\" (UID: \"4f1be45d-006a-4441-98e7-53d6c28c8f57\") "
Aug 04 18:18:17.493286 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:18:17.493214    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f1be45d-006a-4441-98e7-53d6c28c8f57-tmp-dir\") pod \"4f1be45d-006a-4441-98e7-53d6c28c8f57\" (UID: \"4f1be45d-006a-4441-98e7-53d6c28c8f57\") "
Aug 04 18:18:17.493402 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:18:17.493289    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gsm5m\" (UniqueName: \"kubernetes.io/projected/4f1be45d-006a-4441-98e7-53d6c28c8f57-kube-api-access-gsm5m\") pod \"4f1be45d-006a-4441-98e7-53d6c28c8f57\" (UID: \"4f1be45d-006a-4441-98e7-53d6c28c8f57\") "
Aug 04 18:18:17.493402 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:18:17.493329    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f1be45d-006a-4441-98e7-53d6c28c8f57-home\") pod \"4f1be45d-006a-4441-98e7-53d6c28c8f57\" (UID: \"4f1be45d-006a-4441-98e7-53d6c28c8f57\") "
Aug 04 18:18:17.493402 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:18:17.493340    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f1be45d-006a-4441-98e7-53d6c28c8f57-model-cache" (OuterVolumeSpecName: "model-cache") pod "4f1be45d-006a-4441-98e7-53d6c28c8f57" (UID: "4f1be45d-006a-4441-98e7-53d6c28c8f57"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:18:17.493598 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:18:17.493423    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f1be45d-006a-4441-98e7-53d6c28c8f57-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4f1be45d-006a-4441-98e7-53d6c28c8f57" (UID: "4f1be45d-006a-4441-98e7-53d6c28c8f57"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:18:17.493673 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:18:17.493654    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f1be45d-006a-4441-98e7-53d6c28c8f57-home" (OuterVolumeSpecName: "home") pod "4f1be45d-006a-4441-98e7-53d6c28c8f57" (UID: "4f1be45d-006a-4441-98e7-53d6c28c8f57"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:18:17.493735 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:18:17.493685    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f1be45d-006a-4441-98e7-53d6c28c8f57-tmp-dir\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:18:17.493735 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:18:17.493711    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f1be45d-006a-4441-98e7-53d6c28c8f57-model-cache\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:18:17.495324 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:18:17.495282    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f1be45d-006a-4441-98e7-53d6c28c8f57-dshm" (OuterVolumeSpecName: "dshm") pod "4f1be45d-006a-4441-98e7-53d6c28c8f57" (UID: "4f1be45d-006a-4441-98e7-53d6c28c8f57"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:18:17.495416 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:18:17.495355    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f1be45d-006a-4441-98e7-53d6c28c8f57-kube-api-access-gsm5m" (OuterVolumeSpecName: "kube-api-access-gsm5m") pod "4f1be45d-006a-4441-98e7-53d6c28c8f57" (UID: "4f1be45d-006a-4441-98e7-53d6c28c8f57"). InnerVolumeSpecName "kube-api-access-gsm5m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:18:17.495416 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:18:17.495376    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f1be45d-006a-4441-98e7-53d6c28c8f57-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4f1be45d-006a-4441-98e7-53d6c28c8f57" (UID: "4f1be45d-006a-4441-98e7-53d6c28c8f57"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:18:17.595110 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:18:17.595053    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gsm5m\" (UniqueName: \"kubernetes.io/projected/4f1be45d-006a-4441-98e7-53d6c28c8f57-kube-api-access-gsm5m\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:18:17.595110 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:18:17.595074    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f1be45d-006a-4441-98e7-53d6c28c8f57-home\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:18:17.595110 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:18:17.595083    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f1be45d-006a-4441-98e7-53d6c28c8f57-tls-certs\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:18:17.595110 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:18:17.595092    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f1be45d-006a-4441-98e7-53d6c28c8f57-dshm\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:18:17.863867 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:18:17.863791    2583 generic.go:356] "Generic (PLEG): container finished" podID="4f1be45d-006a-4441-98e7-53d6c28c8f57" containerID="b14b2b32e77cc3a59dab029d54afa3d58065fc8e4190d2445b44c9e29275dc62" exitCode=0
Aug 04 18:18:17.863867 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:18:17.863837    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-kd4kg"
Aug 04 18:18:17.863867 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:18:17.863844    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-kd4kg" event={"ID":"4f1be45d-006a-4441-98e7-53d6c28c8f57","Type":"ContainerDied","Data":"b14b2b32e77cc3a59dab029d54afa3d58065fc8e4190d2445b44c9e29275dc62"}
Aug 04 18:18:17.864099 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:18:17.863885    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-kd4kg" event={"ID":"4f1be45d-006a-4441-98e7-53d6c28c8f57","Type":"ContainerDied","Data":"e628e92f53dde8adc4b46fba324ad28e67e29136f44cd3c4f0312e8308fdff1a"}
Aug 04 18:18:17.864099 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:18:17.863907    2583 scope.go:117] "RemoveContainer" containerID="b14b2b32e77cc3a59dab029d54afa3d58065fc8e4190d2445b44c9e29275dc62"
Aug 04 18:18:17.872977 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:18:17.872961    2583 scope.go:117] "RemoveContainer" containerID="b14b2b32e77cc3a59dab029d54afa3d58065fc8e4190d2445b44c9e29275dc62"
Aug 04 18:18:17.873212 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:18:17.873189    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b14b2b32e77cc3a59dab029d54afa3d58065fc8e4190d2445b44c9e29275dc62\": container with ID starting with b14b2b32e77cc3a59dab029d54afa3d58065fc8e4190d2445b44c9e29275dc62 not found: ID does not exist" containerID="b14b2b32e77cc3a59dab029d54afa3d58065fc8e4190d2445b44c9e29275dc62"
Aug 04 18:18:17.873278 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:18:17.873224    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b14b2b32e77cc3a59dab029d54afa3d58065fc8e4190d2445b44c9e29275dc62"} err="failed to get container status \"b14b2b32e77cc3a59dab029d54afa3d58065fc8e4190d2445b44c9e29275dc62\": rpc error: code = NotFound desc = could not find container \"b14b2b32e77cc3a59dab029d54afa3d58065fc8e4190d2445b44c9e29275dc62\": container with ID starting with b14b2b32e77cc3a59dab029d54afa3d58065fc8e4190d2445b44c9e29275dc62 not found: ID does not exist"
Aug 04 18:18:17.904959 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:18:17.904939    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-kd4kg"]
Aug 04 18:18:17.920571 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:18:17.920549    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-kd4kg"]
Aug 04 18:18:18.942392 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:18:18.942352    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4f1be45d-006a-4441-98e7-53d6c28c8f57" path="/var/lib/kubelet/pods/4f1be45d-006a-4441-98e7-53d6c28c8f57/volumes"
Aug 04 18:18:35.681863 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:18:35.681821    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-65dfddc86c-4465g"]
Aug 04 18:18:35.682396 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:18:35.682363    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f1be45d-006a-4441-98e7-53d6c28c8f57" containerName="main"
Aug 04 18:18:35.682396 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:18:35.682383    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f1be45d-006a-4441-98e7-53d6c28c8f57" containerName="main"
Aug 04 18:18:35.682557 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:18:35.682502    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="4f1be45d-006a-4441-98e7-53d6c28c8f57" containerName="main"
Aug 04 18:18:35.687022 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:18:35.686999    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-65dfddc86c-4465g"
Aug 04 18:18:35.695119 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:18:35.695096    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"odh-kserve-custom-ca-bundle\""
Aug 04 18:18:35.695259 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:18:35.695133    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"seaweedfs-s3-creds\""
Aug 04 18:18:35.695396 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:18:35.695378    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"default-dockercfg-c9q9g\""
Aug 04 18:18:35.695683 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:18:35.695666    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"auth-invalid-token-test-kserve-self-signed-certs\""
Aug 04 18:18:35.696184 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:18:35.696166    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"kube-root-ca.crt\""
Aug 04 18:18:35.696271 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:18:35.696174    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"openshift-service-ca.crt\""
Aug 04 18:18:35.708276 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:18:35.708256    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-65dfddc86c-4465g"]
Aug 04 18:18:35.838727 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:18:35.838689    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2swtv\" (UniqueName: \"kubernetes.io/projected/38754199-efa3-4fa9-9c37-e002d88d924c-kube-api-access-2swtv\") pod \"auth-invalid-token-test-kserve-65dfddc86c-4465g\" (UID: \"38754199-efa3-4fa9-9c37-e002d88d924c\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-65dfddc86c-4465g"
Aug 04 18:18:35.838896 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:18:35.838746    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38754199-efa3-4fa9-9c37-e002d88d924c-model-cache\") pod \"auth-invalid-token-test-kserve-65dfddc86c-4465g\" (UID: \"38754199-efa3-4fa9-9c37-e002d88d924c\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-65dfddc86c-4465g"
Aug 04 18:18:35.838896 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:18:35.838802    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/38754199-efa3-4fa9-9c37-e002d88d924c-cabundle-cert\") pod \"auth-invalid-token-test-kserve-65dfddc86c-4465g\" (UID: \"38754199-efa3-4fa9-9c37-e002d88d924c\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-65dfddc86c-4465g"
Aug 04 18:18:35.838896 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:18:35.838852    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38754199-efa3-4fa9-9c37-e002d88d924c-tmp-dir\") pod \"auth-invalid-token-test-kserve-65dfddc86c-4465g\" (UID: \"38754199-efa3-4fa9-9c37-e002d88d924c\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-65dfddc86c-4465g"
Aug 04 18:18:35.838896 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:18:35.838878    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/38754199-efa3-4fa9-9c37-e002d88d924c-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-65dfddc86c-4465g\" (UID: \"38754199-efa3-4fa9-9c37-e002d88d924c\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-65dfddc86c-4465g"
Aug 04 18:18:35.839037 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:18:35.838929    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/38754199-efa3-4fa9-9c37-e002d88d924c-home\") pod \"auth-invalid-token-test-kserve-65dfddc86c-4465g\" (UID: \"38754199-efa3-4fa9-9c37-e002d88d924c\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-65dfddc86c-4465g"
Aug 04 18:18:35.839037 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:18:35.838948    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/38754199-efa3-4fa9-9c37-e002d88d924c-dshm\") pod \"auth-invalid-token-test-kserve-65dfddc86c-4465g\" (UID: \"38754199-efa3-4fa9-9c37-e002d88d924c\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-65dfddc86c-4465g"
Aug 04 18:18:35.839037 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:18:35.838972    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38754199-efa3-4fa9-9c37-e002d88d924c-tls-certs\") pod \"auth-invalid-token-test-kserve-65dfddc86c-4465g\" (UID: \"38754199-efa3-4fa9-9c37-e002d88d924c\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-65dfddc86c-4465g"
Aug 04 18:18:35.940299 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:18:35.940263    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38754199-efa3-4fa9-9c37-e002d88d924c-model-cache\") pod \"auth-invalid-token-test-kserve-65dfddc86c-4465g\" (UID: \"38754199-efa3-4fa9-9c37-e002d88d924c\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-65dfddc86c-4465g"
Aug 04 18:18:35.940522 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:18:35.940304    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/38754199-efa3-4fa9-9c37-e002d88d924c-cabundle-cert\") pod \"auth-invalid-token-test-kserve-65dfddc86c-4465g\" (UID: \"38754199-efa3-4fa9-9c37-e002d88d924c\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-65dfddc86c-4465g"
Aug 04 18:18:35.940522 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:18:35.940334    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38754199-efa3-4fa9-9c37-e002d88d924c-tmp-dir\") pod \"auth-invalid-token-test-kserve-65dfddc86c-4465g\" (UID: \"38754199-efa3-4fa9-9c37-e002d88d924c\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-65dfddc86c-4465g"
Aug 04 18:18:35.940522 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:18:35.940354    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/38754199-efa3-4fa9-9c37-e002d88d924c-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-65dfddc86c-4465g\" (UID: \"38754199-efa3-4fa9-9c37-e002d88d924c\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-65dfddc86c-4465g"
Aug 04 18:18:35.940522 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:18:35.940383    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/38754199-efa3-4fa9-9c37-e002d88d924c-home\") pod \"auth-invalid-token-test-kserve-65dfddc86c-4465g\" (UID: \"38754199-efa3-4fa9-9c37-e002d88d924c\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-65dfddc86c-4465g"
Aug 04 18:18:35.940522 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:18:35.940412    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/38754199-efa3-4fa9-9c37-e002d88d924c-dshm\") pod \"auth-invalid-token-test-kserve-65dfddc86c-4465g\" (UID: \"38754199-efa3-4fa9-9c37-e002d88d924c\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-65dfddc86c-4465g"
Aug 04 18:18:35.940522 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:18:35.940427    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38754199-efa3-4fa9-9c37-e002d88d924c-tls-certs\") pod \"auth-invalid-token-test-kserve-65dfddc86c-4465g\" (UID: \"38754199-efa3-4fa9-9c37-e002d88d924c\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-65dfddc86c-4465g"
Aug 04 18:18:35.940522 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:18:35.940463    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2swtv\" (UniqueName: \"kubernetes.io/projected/38754199-efa3-4fa9-9c37-e002d88d924c-kube-api-access-2swtv\") pod \"auth-invalid-token-test-kserve-65dfddc86c-4465g\" (UID: \"38754199-efa3-4fa9-9c37-e002d88d924c\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-65dfddc86c-4465g"
Aug 04 18:18:35.940873 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:18:35.940823    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38754199-efa3-4fa9-9c37-e002d88d924c-tmp-dir\") pod \"auth-invalid-token-test-kserve-65dfddc86c-4465g\" (UID: \"38754199-efa3-4fa9-9c37-e002d88d924c\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-65dfddc86c-4465g"
Aug 04 18:18:35.940933 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:18:35.940874    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/38754199-efa3-4fa9-9c37-e002d88d924c-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-65dfddc86c-4465g\" (UID: \"38754199-efa3-4fa9-9c37-e002d88d924c\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-65dfddc86c-4465g"
Aug 04 18:18:35.940933 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:18:35.940906    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/38754199-efa3-4fa9-9c37-e002d88d924c-home\") pod \"auth-invalid-token-test-kserve-65dfddc86c-4465g\" (UID: \"38754199-efa3-4fa9-9c37-e002d88d924c\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-65dfddc86c-4465g"
Aug 04 18:18:35.941029 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:18:35.940967    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38754199-efa3-4fa9-9c37-e002d88d924c-model-cache\") pod \"auth-invalid-token-test-kserve-65dfddc86c-4465g\" (UID: \"38754199-efa3-4fa9-9c37-e002d88d924c\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-65dfddc86c-4465g"
Aug 04 18:18:35.941151 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:18:35.941128    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/38754199-efa3-4fa9-9c37-e002d88d924c-cabundle-cert\") pod \"auth-invalid-token-test-kserve-65dfddc86c-4465g\" (UID: \"38754199-efa3-4fa9-9c37-e002d88d924c\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-65dfddc86c-4465g"
Aug 04 18:18:35.942814 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:18:35.942788    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/38754199-efa3-4fa9-9c37-e002d88d924c-dshm\") pod \"auth-invalid-token-test-kserve-65dfddc86c-4465g\" (UID: \"38754199-efa3-4fa9-9c37-e002d88d924c\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-65dfddc86c-4465g"
Aug 04 18:18:35.943096 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:18:35.943079    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38754199-efa3-4fa9-9c37-e002d88d924c-tls-certs\") pod \"auth-invalid-token-test-kserve-65dfddc86c-4465g\" (UID: \"38754199-efa3-4fa9-9c37-e002d88d924c\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-65dfddc86c-4465g"
Aug 04 18:18:35.959746 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:18:35.959717    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2swtv\" (UniqueName: \"kubernetes.io/projected/38754199-efa3-4fa9-9c37-e002d88d924c-kube-api-access-2swtv\") pod \"auth-invalid-token-test-kserve-65dfddc86c-4465g\" (UID: \"38754199-efa3-4fa9-9c37-e002d88d924c\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-65dfddc86c-4465g"
Aug 04 18:18:35.997758 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:18:35.997725    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-65dfddc86c-4465g"
Aug 04 18:18:36.150779 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:18:36.150751    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-65dfddc86c-4465g"]
Aug 04 18:18:36.151318 ip-10-0-135-124 kubenswrapper[2583]: W0804 18:18:36.151289    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38754199_efa3_4fa9_9c37_e002d88d924c.slice/crio-12de9b3c6d61f395a0398dbb1d432168f3290c87eb58043b89d29b7b6b617ad3 WatchSource:0}: Error finding container 12de9b3c6d61f395a0398dbb1d432168f3290c87eb58043b89d29b7b6b617ad3: Status 404 returned error can't find the container with id 12de9b3c6d61f395a0398dbb1d432168f3290c87eb58043b89d29b7b6b617ad3
Aug 04 18:18:36.971427 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:18:36.971388    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-65dfddc86c-4465g" event={"ID":"38754199-efa3-4fa9-9c37-e002d88d924c","Type":"ContainerStarted","Data":"2b15d933c6f3bcdc907d8a19c54ac4c1f79d3d7ebbb9b20c7bf08ed0c5fdfbe6"}
Aug 04 18:18:36.971933 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:18:36.971457    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-65dfddc86c-4465g" event={"ID":"38754199-efa3-4fa9-9c37-e002d88d924c","Type":"ContainerStarted","Data":"12de9b3c6d61f395a0398dbb1d432168f3290c87eb58043b89d29b7b6b617ad3"}
Aug 04 18:18:38.983913 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:18:38.983877    2583 generic.go:356] "Generic (PLEG): container finished" podID="38754199-efa3-4fa9-9c37-e002d88d924c" containerID="2b15d933c6f3bcdc907d8a19c54ac4c1f79d3d7ebbb9b20c7bf08ed0c5fdfbe6" exitCode=0
Aug 04 18:18:38.984291 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:18:38.983950    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-65dfddc86c-4465g" event={"ID":"38754199-efa3-4fa9-9c37-e002d88d924c","Type":"ContainerDied","Data":"2b15d933c6f3bcdc907d8a19c54ac4c1f79d3d7ebbb9b20c7bf08ed0c5fdfbe6"}
Aug 04 18:18:39.991441 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:18:39.991407    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-65dfddc86c-4465g" event={"ID":"38754199-efa3-4fa9-9c37-e002d88d924c","Type":"ContainerStarted","Data":"97cece9305dfe90207cf7c7a88f6799136ae6c5879cb359ddea4d88dd297b6d6"}
Aug 04 18:18:39.997982 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:18:39.997963    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-65dfddc86c-4465g"
Aug 04 18:18:40.014307 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:18:40.014260    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-65dfddc86c-4465g" podStartSLOduration=2.90759684 podStartE2EDuration="5.014247953s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.106651113s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:18:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:18:40.013663138 +0000 UTC m=+1133.594881717" watchObservedRunningTime="2026-08-04 18:18:40.014247953 +0000 UTC m=+1133.595466530"
Aug 04 18:18:45.998150 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:18:45.998112    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-65dfddc86c-4465g"
Aug 04 18:18:45.999972 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:18:45.999943    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-65dfddc86c-4465g" podUID="38754199-efa3-4fa9-9c37-e002d88d924c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": dial tcp 10.132.0.53:8000: connect: connection refused"
Aug 04 18:18:46.995169 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:18:46.995141    2583 scope.go:117] "RemoveContainer" containerID="bceb32f64c846297b1ad3664ab26b8395375ae891b807360829cef37a66a935f"
Aug 04 18:18:47.003650 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:18:47.003630    2583 scope.go:117] "RemoveContainer" containerID="c223ffa472129e8a00afa09a1033f063b937fd225bf514b8d7785d871e7ca2f8"
Aug 04 18:18:55.999033 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:18:55.998981    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-65dfddc86c-4465g" podUID="38754199-efa3-4fa9-9c37-e002d88d924c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": dial tcp 10.132.0.53:8000: connect: connection refused"
Aug 04 18:19:00.959416 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:19:00.959368    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf4tm8c"]
Aug 04 18:19:00.962556 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:19:00.962535    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf4tm8c"
Aug 04 18:19:00.967448 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:19:00.967424    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"default-dockercfg-lhf68\""
Aug 04 18:19:00.967576 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:19:00.967483    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"scheduler-ha-replicas-test-kserve-self-signed-certs\""
Aug 04 18:19:00.967576 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:19:00.967505    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"scheduler-ha-replicas-test-epp-sa-dockercfg-ztqzn\""
Aug 04 18:19:00.967576 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:19:00.967528    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"openshift-service-ca.crt\""
Aug 04 18:19:00.967576 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:19:00.967424    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"kube-root-ca.crt\""
Aug 04 18:19:00.972984 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:19:00.972963    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf4tm8c"]
Aug 04 18:19:01.051448 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:19:01.051411    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e33360b-2ede-4d3f-b18d-e4cf53892536-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf4tm8c\" (UID: \"2e33360b-2ede-4d3f-b18d-e4cf53892536\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf4tm8c"
Aug 04 18:19:01.051660 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:19:01.051554    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ls9t\" (UniqueName: \"kubernetes.io/projected/2e33360b-2ede-4d3f-b18d-e4cf53892536-kube-api-access-8ls9t\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf4tm8c\" (UID: \"2e33360b-2ede-4d3f-b18d-e4cf53892536\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf4tm8c"
Aug 04 18:19:01.152997 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:19:01.152962    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e33360b-2ede-4d3f-b18d-e4cf53892536-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf4tm8c\" (UID: \"2e33360b-2ede-4d3f-b18d-e4cf53892536\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf4tm8c"
Aug 04 18:19:01.153169 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:19:01.153043    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ls9t\" (UniqueName: \"kubernetes.io/projected/2e33360b-2ede-4d3f-b18d-e4cf53892536-kube-api-access-8ls9t\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf4tm8c\" (UID: \"2e33360b-2ede-4d3f-b18d-e4cf53892536\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf4tm8c"
Aug 04 18:19:01.155679 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:19:01.155642    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e33360b-2ede-4d3f-b18d-e4cf53892536-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf4tm8c\" (UID: \"2e33360b-2ede-4d3f-b18d-e4cf53892536\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf4tm8c"
Aug 04 18:19:01.162862 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:19:01.162835    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ls9t\" (UniqueName: \"kubernetes.io/projected/2e33360b-2ede-4d3f-b18d-e4cf53892536-kube-api-access-8ls9t\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf4tm8c\" (UID: \"2e33360b-2ede-4d3f-b18d-e4cf53892536\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf4tm8c"
Aug 04 18:19:01.273923 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:19:01.273844    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf4tm8c"
Aug 04 18:19:01.413237 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:19:01.413204    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf4tm8c"]
Aug 04 18:19:01.413538 ip-10-0-135-124 kubenswrapper[2583]: W0804 18:19:01.413511    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e33360b_2ede_4d3f_b18d_e4cf53892536.slice/crio-60c687cbf15a6e68ed30e2bb92b99bf830f97feff5f5039b8db80561b7afed85 WatchSource:0}: Error finding container 60c687cbf15a6e68ed30e2bb92b99bf830f97feff5f5039b8db80561b7afed85: Status 404 returned error can't find the container with id 60c687cbf15a6e68ed30e2bb92b99bf830f97feff5f5039b8db80561b7afed85
Aug 04 18:19:02.114129 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:19:02.114097    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf4tm8c" event={"ID":"2e33360b-2ede-4d3f-b18d-e4cf53892536","Type":"ContainerStarted","Data":"c703bd4c9fdf82f594c0254800943c263a30edf1bd4a36497a9ccc67124edd00"}
Aug 04 18:19:02.114600 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:19:02.114154    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf4tm8c" event={"ID":"2e33360b-2ede-4d3f-b18d-e4cf53892536","Type":"ContainerStarted","Data":"60c687cbf15a6e68ed30e2bb92b99bf830f97feff5f5039b8db80561b7afed85"}
Aug 04 18:19:02.114600 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:19:02.114220    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf4tm8c"
Aug 04 18:19:02.135413 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:19:02.135368    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf4tm8c" podStartSLOduration=2.135352972 podStartE2EDuration="2.135352972s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:19:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:19:02.13219186 +0000 UTC m=+1155.713410460" watchObservedRunningTime="2026-08-04 18:19:02.135352972 +0000 UTC m=+1155.716571549"
Aug 04 18:19:05.998562 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:19:05.998524    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-65dfddc86c-4465g" podUID="38754199-efa3-4fa9-9c37-e002d88d924c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": dial tcp 10.132.0.53:8000: connect: connection refused"
Aug 04 18:19:15.999097 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:19:15.999009    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-65dfddc86c-4465g" podUID="38754199-efa3-4fa9-9c37-e002d88d924c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": dial tcp 10.132.0.53:8000: connect: connection refused"
Aug 04 18:19:25.998661 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:19:25.998616    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-65dfddc86c-4465g" podUID="38754199-efa3-4fa9-9c37-e002d88d924c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": dial tcp 10.132.0.53:8000: connect: connection refused"
Aug 04 18:19:33.121562 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:19:33.121521    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf4tm8c" podUID="2e33360b-2ede-4d3f-b18d-e4cf53892536" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Aug 04 18:19:34.515945 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:19:34.515905    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf4tm8c"]
Aug 04 18:19:34.518449 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:19:34.518414    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf4tm8c" podUID="2e33360b-2ede-4d3f-b18d-e4cf53892536" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Aug 04 18:19:35.998548 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:19:35.998502    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-65dfddc86c-4465g" podUID="38754199-efa3-4fa9-9c37-e002d88d924c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": dial tcp 10.132.0.53:8000: connect: connection refused"
Aug 04 18:19:44.518427 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:19:44.518393    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf4tm8c" podUID="2e33360b-2ede-4d3f-b18d-e4cf53892536" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Aug 04 18:19:45.998890 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:19:45.998844    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-65dfddc86c-4465g" podUID="38754199-efa3-4fa9-9c37-e002d88d924c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": dial tcp 10.132.0.53:8000: connect: connection refused"
Aug 04 18:19:46.959659 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:19:46.959632    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-wskfb_6157c2d3-2fd0-4544-88ad-434ac75f8c04/console-operator/1.log"
Aug 04 18:19:46.963185 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:19:46.963165    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-wskfb_6157c2d3-2fd0-4544-88ad-434ac75f8c04/console-operator/1.log"
Aug 04 18:19:49.538382 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:19:49.538320    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf4tm8c" podUID="2e33360b-2ede-4d3f-b18d-e4cf53892536" containerName="main" containerID="cri-o://c703bd4c9fdf82f594c0254800943c263a30edf1bd4a36497a9ccc67124edd00" gracePeriod=45
Aug 04 18:19:49.794662 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:19:49.794597    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf4tm8c"
Aug 04 18:19:49.883268 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:19:49.883230    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8ls9t\" (UniqueName: \"kubernetes.io/projected/2e33360b-2ede-4d3f-b18d-e4cf53892536-kube-api-access-8ls9t\") pod \"2e33360b-2ede-4d3f-b18d-e4cf53892536\" (UID: \"2e33360b-2ede-4d3f-b18d-e4cf53892536\") "
Aug 04 18:19:49.883268 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:19:49.883269    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e33360b-2ede-4d3f-b18d-e4cf53892536-tls-certs\") pod \"2e33360b-2ede-4d3f-b18d-e4cf53892536\" (UID: \"2e33360b-2ede-4d3f-b18d-e4cf53892536\") "
Aug 04 18:19:49.885485 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:19:49.885430    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e33360b-2ede-4d3f-b18d-e4cf53892536-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2e33360b-2ede-4d3f-b18d-e4cf53892536" (UID: "2e33360b-2ede-4d3f-b18d-e4cf53892536"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:19:49.885603 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:19:49.885533    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e33360b-2ede-4d3f-b18d-e4cf53892536-kube-api-access-8ls9t" (OuterVolumeSpecName: "kube-api-access-8ls9t") pod "2e33360b-2ede-4d3f-b18d-e4cf53892536" (UID: "2e33360b-2ede-4d3f-b18d-e4cf53892536"). InnerVolumeSpecName "kube-api-access-8ls9t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:19:49.984870 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:19:49.984837    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8ls9t\" (UniqueName: \"kubernetes.io/projected/2e33360b-2ede-4d3f-b18d-e4cf53892536-kube-api-access-8ls9t\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:19:49.984870 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:19:49.984870    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e33360b-2ede-4d3f-b18d-e4cf53892536-tls-certs\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:19:50.388459 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:19:50.388420    2583 generic.go:356] "Generic (PLEG): container finished" podID="2e33360b-2ede-4d3f-b18d-e4cf53892536" containerID="c703bd4c9fdf82f594c0254800943c263a30edf1bd4a36497a9ccc67124edd00" exitCode=0
Aug 04 18:19:50.388654 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:19:50.388461    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf4tm8c"
Aug 04 18:19:50.388654 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:19:50.388498    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf4tm8c" event={"ID":"2e33360b-2ede-4d3f-b18d-e4cf53892536","Type":"ContainerDied","Data":"c703bd4c9fdf82f594c0254800943c263a30edf1bd4a36497a9ccc67124edd00"}
Aug 04 18:19:50.388654 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:19:50.388556    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf4tm8c" event={"ID":"2e33360b-2ede-4d3f-b18d-e4cf53892536","Type":"ContainerDied","Data":"60c687cbf15a6e68ed30e2bb92b99bf830f97feff5f5039b8db80561b7afed85"}
Aug 04 18:19:50.388654 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:19:50.388596    2583 scope.go:117] "RemoveContainer" containerID="c703bd4c9fdf82f594c0254800943c263a30edf1bd4a36497a9ccc67124edd00"
Aug 04 18:19:50.399264 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:19:50.399243    2583 scope.go:117] "RemoveContainer" containerID="c703bd4c9fdf82f594c0254800943c263a30edf1bd4a36497a9ccc67124edd00"
Aug 04 18:19:50.399772 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:19:50.399747    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c703bd4c9fdf82f594c0254800943c263a30edf1bd4a36497a9ccc67124edd00\": container with ID starting with c703bd4c9fdf82f594c0254800943c263a30edf1bd4a36497a9ccc67124edd00 not found: ID does not exist" containerID="c703bd4c9fdf82f594c0254800943c263a30edf1bd4a36497a9ccc67124edd00"
Aug 04 18:19:50.399932 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:19:50.399910    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c703bd4c9fdf82f594c0254800943c263a30edf1bd4a36497a9ccc67124edd00"} err="failed to get container status \"c703bd4c9fdf82f594c0254800943c263a30edf1bd4a36497a9ccc67124edd00\": rpc error: code = NotFound desc = could not find container \"c703bd4c9fdf82f594c0254800943c263a30edf1bd4a36497a9ccc67124edd00\": container with ID starting with c703bd4c9fdf82f594c0254800943c263a30edf1bd4a36497a9ccc67124edd00 not found: ID does not exist"
Aug 04 18:19:50.411045 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:19:50.411017    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf4tm8c"]
Aug 04 18:19:50.414580 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:19:50.414556    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cf4tm8c"]
Aug 04 18:19:50.943165 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:19:50.943131    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2e33360b-2ede-4d3f-b18d-e4cf53892536" path="/var/lib/kubelet/pods/2e33360b-2ede-4d3f-b18d-e4cf53892536/volumes"
Aug 04 18:19:55.999037 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:19:55.998999    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-65dfddc86c-4465g" podUID="38754199-efa3-4fa9-9c37-e002d88d924c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": dial tcp 10.132.0.53:8000: connect: connection refused"
Aug 04 18:20:05.998843 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:20:05.998802    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-65dfddc86c-4465g" podUID="38754199-efa3-4fa9-9c37-e002d88d924c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": dial tcp 10.132.0.53:8000: connect: connection refused"
Aug 04 18:20:15.998245 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:20:15.998197    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-65dfddc86c-4465g" podUID="38754199-efa3-4fa9-9c37-e002d88d924c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": dial tcp 10.132.0.53:8000: connect: connection refused"
Aug 04 18:20:25.998895 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:20:25.998836    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-65dfddc86c-4465g" podUID="38754199-efa3-4fa9-9c37-e002d88d924c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": dial tcp 10.132.0.53:8000: connect: connection refused"
Aug 04 18:20:36.008886 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:20:36.008845    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-65dfddc86c-4465g"
Aug 04 18:20:36.016835 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:20:36.016814    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-65dfddc86c-4465g"
Aug 04 18:20:37.534888 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:20:37.534856    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-65dfddc86c-4465g"]
Aug 04 18:20:52.693738 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:20:52.693692    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-65dfddc86c-4465g" podUID="38754199-efa3-4fa9-9c37-e002d88d924c" containerName="main" containerID="cri-o://97cece9305dfe90207cf7c7a88f6799136ae6c5879cb359ddea4d88dd297b6d6" gracePeriod=45
Aug 04 18:20:53.009581 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:20:53.009460    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-65dfddc86c-4465g" podUID="38754199-efa3-4fa9-9c37-e002d88d924c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": dial tcp 10.132.0.53:8000: connect: connection refused"
Aug 04 18:20:53.446852 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:20:53.446828    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-65dfddc86c-4465g"
Aug 04 18:20:53.530825 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:20:53.530793    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/38754199-efa3-4fa9-9c37-e002d88d924c-cabundle-cert\") pod \"38754199-efa3-4fa9-9c37-e002d88d924c\" (UID: \"38754199-efa3-4fa9-9c37-e002d88d924c\") "
Aug 04 18:20:53.531006 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:20:53.530844    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2swtv\" (UniqueName: \"kubernetes.io/projected/38754199-efa3-4fa9-9c37-e002d88d924c-kube-api-access-2swtv\") pod \"38754199-efa3-4fa9-9c37-e002d88d924c\" (UID: \"38754199-efa3-4fa9-9c37-e002d88d924c\") "
Aug 04 18:20:53.531006 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:20:53.530895    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/38754199-efa3-4fa9-9c37-e002d88d924c-home\") pod \"38754199-efa3-4fa9-9c37-e002d88d924c\" (UID: \"38754199-efa3-4fa9-9c37-e002d88d924c\") "
Aug 04 18:20:53.531006 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:20:53.530928    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/38754199-efa3-4fa9-9c37-e002d88d924c-kserve-provision-location\") pod \"38754199-efa3-4fa9-9c37-e002d88d924c\" (UID: \"38754199-efa3-4fa9-9c37-e002d88d924c\") "
Aug 04 18:20:53.531006 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:20:53.530971    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38754199-efa3-4fa9-9c37-e002d88d924c-model-cache\") pod \"38754199-efa3-4fa9-9c37-e002d88d924c\" (UID: \"38754199-efa3-4fa9-9c37-e002d88d924c\") "
Aug 04 18:20:53.531244 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:20:53.531024    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/38754199-efa3-4fa9-9c37-e002d88d924c-dshm\") pod \"38754199-efa3-4fa9-9c37-e002d88d924c\" (UID: \"38754199-efa3-4fa9-9c37-e002d88d924c\") "
Aug 04 18:20:53.531244 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:20:53.531052    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38754199-efa3-4fa9-9c37-e002d88d924c-tmp-dir\") pod \"38754199-efa3-4fa9-9c37-e002d88d924c\" (UID: \"38754199-efa3-4fa9-9c37-e002d88d924c\") "
Aug 04 18:20:53.531244 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:20:53.531099    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38754199-efa3-4fa9-9c37-e002d88d924c-tls-certs\") pod \"38754199-efa3-4fa9-9c37-e002d88d924c\" (UID: \"38754199-efa3-4fa9-9c37-e002d88d924c\") "
Aug 04 18:20:53.531400 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:20:53.531239    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38754199-efa3-4fa9-9c37-e002d88d924c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "38754199-efa3-4fa9-9c37-e002d88d924c" (UID: "38754199-efa3-4fa9-9c37-e002d88d924c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:20:53.531400 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:20:53.531311    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38754199-efa3-4fa9-9c37-e002d88d924c-model-cache" (OuterVolumeSpecName: "model-cache") pod "38754199-efa3-4fa9-9c37-e002d88d924c" (UID: "38754199-efa3-4fa9-9c37-e002d88d924c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:20:53.531544 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:20:53.531451    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38754199-efa3-4fa9-9c37-e002d88d924c-model-cache\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:20:53.531544 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:20:53.531492    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/38754199-efa3-4fa9-9c37-e002d88d924c-cabundle-cert\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:20:53.531737 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:20:53.531714    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38754199-efa3-4fa9-9c37-e002d88d924c-home" (OuterVolumeSpecName: "home") pod "38754199-efa3-4fa9-9c37-e002d88d924c" (UID: "38754199-efa3-4fa9-9c37-e002d88d924c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:20:53.533806 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:20:53.533767    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38754199-efa3-4fa9-9c37-e002d88d924c-kube-api-access-2swtv" (OuterVolumeSpecName: "kube-api-access-2swtv") pod "38754199-efa3-4fa9-9c37-e002d88d924c" (UID: "38754199-efa3-4fa9-9c37-e002d88d924c"). InnerVolumeSpecName "kube-api-access-2swtv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:20:53.533927 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:20:53.533886    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/38754199-efa3-4fa9-9c37-e002d88d924c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "38754199-efa3-4fa9-9c37-e002d88d924c" (UID: "38754199-efa3-4fa9-9c37-e002d88d924c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:20:53.535045 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:20:53.535012    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38754199-efa3-4fa9-9c37-e002d88d924c-dshm" (OuterVolumeSpecName: "dshm") pod "38754199-efa3-4fa9-9c37-e002d88d924c" (UID: "38754199-efa3-4fa9-9c37-e002d88d924c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:20:53.544121 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:20:53.544091    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38754199-efa3-4fa9-9c37-e002d88d924c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "38754199-efa3-4fa9-9c37-e002d88d924c" (UID: "38754199-efa3-4fa9-9c37-e002d88d924c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:20:53.553543 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:20:53.553516    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38754199-efa3-4fa9-9c37-e002d88d924c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "38754199-efa3-4fa9-9c37-e002d88d924c" (UID: "38754199-efa3-4fa9-9c37-e002d88d924c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:20:53.632235 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:20:53.632146    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38754199-efa3-4fa9-9c37-e002d88d924c-tls-certs\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:20:53.632235 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:20:53.632179    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2swtv\" (UniqueName: \"kubernetes.io/projected/38754199-efa3-4fa9-9c37-e002d88d924c-kube-api-access-2swtv\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:20:53.632235 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:20:53.632188    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/38754199-efa3-4fa9-9c37-e002d88d924c-home\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:20:53.632235 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:20:53.632198    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/38754199-efa3-4fa9-9c37-e002d88d924c-kserve-provision-location\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:20:53.632235 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:20:53.632207    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/38754199-efa3-4fa9-9c37-e002d88d924c-dshm\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:20:53.632235 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:20:53.632215    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38754199-efa3-4fa9-9c37-e002d88d924c-tmp-dir\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:20:53.759985 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:20:53.759950    2583 generic.go:356] "Generic (PLEG): container finished" podID="38754199-efa3-4fa9-9c37-e002d88d924c" containerID="97cece9305dfe90207cf7c7a88f6799136ae6c5879cb359ddea4d88dd297b6d6" exitCode=0
Aug 04 18:20:53.760360 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:20:53.760000    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-65dfddc86c-4465g"
Aug 04 18:20:53.760360 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:20:53.760008    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-65dfddc86c-4465g" event={"ID":"38754199-efa3-4fa9-9c37-e002d88d924c","Type":"ContainerDied","Data":"97cece9305dfe90207cf7c7a88f6799136ae6c5879cb359ddea4d88dd297b6d6"}
Aug 04 18:20:53.760360 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:20:53.760052    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-65dfddc86c-4465g" event={"ID":"38754199-efa3-4fa9-9c37-e002d88d924c","Type":"ContainerDied","Data":"12de9b3c6d61f395a0398dbb1d432168f3290c87eb58043b89d29b7b6b617ad3"}
Aug 04 18:20:53.760360 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:20:53.760073    2583 scope.go:117] "RemoveContainer" containerID="97cece9305dfe90207cf7c7a88f6799136ae6c5879cb359ddea4d88dd297b6d6"
Aug 04 18:20:53.769721 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:20:53.769673    2583 scope.go:117] "RemoveContainer" containerID="2b15d933c6f3bcdc907d8a19c54ac4c1f79d3d7ebbb9b20c7bf08ed0c5fdfbe6"
Aug 04 18:20:53.777936 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:20:53.777909    2583 scope.go:117] "RemoveContainer" containerID="97cece9305dfe90207cf7c7a88f6799136ae6c5879cb359ddea4d88dd297b6d6"
Aug 04 18:20:53.778171 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:20:53.778152    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97cece9305dfe90207cf7c7a88f6799136ae6c5879cb359ddea4d88dd297b6d6\": container with ID starting with 97cece9305dfe90207cf7c7a88f6799136ae6c5879cb359ddea4d88dd297b6d6 not found: ID does not exist" containerID="97cece9305dfe90207cf7c7a88f6799136ae6c5879cb359ddea4d88dd297b6d6"
Aug 04 18:20:53.778229 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:20:53.778179    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97cece9305dfe90207cf7c7a88f6799136ae6c5879cb359ddea4d88dd297b6d6"} err="failed to get container status \"97cece9305dfe90207cf7c7a88f6799136ae6c5879cb359ddea4d88dd297b6d6\": rpc error: code = NotFound desc = could not find container \"97cece9305dfe90207cf7c7a88f6799136ae6c5879cb359ddea4d88dd297b6d6\": container with ID starting with 97cece9305dfe90207cf7c7a88f6799136ae6c5879cb359ddea4d88dd297b6d6 not found: ID does not exist"
Aug 04 18:20:53.778229 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:20:53.778202    2583 scope.go:117] "RemoveContainer" containerID="2b15d933c6f3bcdc907d8a19c54ac4c1f79d3d7ebbb9b20c7bf08ed0c5fdfbe6"
Aug 04 18:20:53.778412 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:20:53.778394    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b15d933c6f3bcdc907d8a19c54ac4c1f79d3d7ebbb9b20c7bf08ed0c5fdfbe6\": container with ID starting with 2b15d933c6f3bcdc907d8a19c54ac4c1f79d3d7ebbb9b20c7bf08ed0c5fdfbe6 not found: ID does not exist" containerID="2b15d933c6f3bcdc907d8a19c54ac4c1f79d3d7ebbb9b20c7bf08ed0c5fdfbe6"
Aug 04 18:20:53.778456 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:20:53.778416    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b15d933c6f3bcdc907d8a19c54ac4c1f79d3d7ebbb9b20c7bf08ed0c5fdfbe6"} err="failed to get container status \"2b15d933c6f3bcdc907d8a19c54ac4c1f79d3d7ebbb9b20c7bf08ed0c5fdfbe6\": rpc error: code = NotFound desc = could not find container \"2b15d933c6f3bcdc907d8a19c54ac4c1f79d3d7ebbb9b20c7bf08ed0c5fdfbe6\": container with ID starting with 2b15d933c6f3bcdc907d8a19c54ac4c1f79d3d7ebbb9b20c7bf08ed0c5fdfbe6 not found: ID does not exist"
Aug 04 18:20:53.785615 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:20:53.785593    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-65dfddc86c-4465g"]
Aug 04 18:20:53.789151 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:20:53.789129    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-65dfddc86c-4465g"]
Aug 04 18:20:54.941965 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:20:54.941921    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="38754199-efa3-4fa9-9c37-e002d88d924c" path="/var/lib/kubelet/pods/38754199-efa3-4fa9-9c37-e002d88d924c/volumes"
Aug 04 18:21:05.219893 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:21:05.219860    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f9jftff"]
Aug 04 18:21:05.220366 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:21:05.220216    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38754199-efa3-4fa9-9c37-e002d88d924c" containerName="storage-initializer"
Aug 04 18:21:05.220366 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:21:05.220229    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="38754199-efa3-4fa9-9c37-e002d88d924c" containerName="storage-initializer"
Aug 04 18:21:05.220366 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:21:05.220249    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38754199-efa3-4fa9-9c37-e002d88d924c" containerName="main"
Aug 04 18:21:05.220366 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:21:05.220255    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="38754199-efa3-4fa9-9c37-e002d88d924c" containerName="main"
Aug 04 18:21:05.220366 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:21:05.220271    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e33360b-2ede-4d3f-b18d-e4cf53892536" containerName="main"
Aug 04 18:21:05.220366 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:21:05.220277    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e33360b-2ede-4d3f-b18d-e4cf53892536" containerName="main"
Aug 04 18:21:05.220366 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:21:05.220334    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="38754199-efa3-4fa9-9c37-e002d88d924c" containerName="main"
Aug 04 18:21:05.220366 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:21:05.220344    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e33360b-2ede-4d3f-b18d-e4cf53892536" containerName="main"
Aug 04 18:21:05.224913 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:21:05.224890    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f9jftff"
Aug 04 18:21:05.227797 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:21:05.227770    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"openshift-service-ca.crt\""
Aug 04 18:21:05.227913 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:21:05.227888    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"default-dockercfg-xdbl6\""
Aug 04 18:21:05.229224 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:21:05.229201    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"kube-root-ca.crt\""
Aug 04 18:21:05.229521 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:21:05.229502    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"scheduler-v06-pd-migration-test-epp-sa-dockercfg-x4rsj\""
Aug 04 18:21:05.229713 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:21:05.229517    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"scheduler-v06-pd-migration-test-kserve-self-signed-certs\""
Aug 04 18:21:05.233496 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:21:05.233433    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f9jftff"]
Aug 04 18:21:05.337220 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:21:05.337182    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xhlj\" (UniqueName: \"kubernetes.io/projected/2256be5b-f6a8-4be7-b774-27c21ddfa89f-kube-api-access-5xhlj\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-f9jftff\" (UID: \"2256be5b-f6a8-4be7-b774-27c21ddfa89f\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f9jftff"
Aug 04 18:21:05.337387 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:21:05.337245    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2256be5b-f6a8-4be7-b774-27c21ddfa89f-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-f9jftff\" (UID: \"2256be5b-f6a8-4be7-b774-27c21ddfa89f\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f9jftff"
Aug 04 18:21:05.438747 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:21:05.438713    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xhlj\" (UniqueName: \"kubernetes.io/projected/2256be5b-f6a8-4be7-b774-27c21ddfa89f-kube-api-access-5xhlj\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-f9jftff\" (UID: \"2256be5b-f6a8-4be7-b774-27c21ddfa89f\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f9jftff"
Aug 04 18:21:05.438924 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:21:05.438768    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2256be5b-f6a8-4be7-b774-27c21ddfa89f-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-f9jftff\" (UID: \"2256be5b-f6a8-4be7-b774-27c21ddfa89f\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f9jftff"
Aug 04 18:21:05.441285 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:21:05.441257    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2256be5b-f6a8-4be7-b774-27c21ddfa89f-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-f9jftff\" (UID: \"2256be5b-f6a8-4be7-b774-27c21ddfa89f\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f9jftff"
Aug 04 18:21:05.446588 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:21:05.446565    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xhlj\" (UniqueName: \"kubernetes.io/projected/2256be5b-f6a8-4be7-b774-27c21ddfa89f-kube-api-access-5xhlj\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-f9jftff\" (UID: \"2256be5b-f6a8-4be7-b774-27c21ddfa89f\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f9jftff"
Aug 04 18:21:05.538837 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:21:05.538750    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f9jftff"
Aug 04 18:21:05.681297 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:21:05.681269    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f9jftff"]
Aug 04 18:21:05.683279 ip-10-0-135-124 kubenswrapper[2583]: W0804 18:21:05.683252    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2256be5b_f6a8_4be7_b774_27c21ddfa89f.slice/crio-3b563c0379586a7343c34300d5e9a6b9e17b1262e823e8a6d23d4617a7f2f0d1 WatchSource:0}: Error finding container 3b563c0379586a7343c34300d5e9a6b9e17b1262e823e8a6d23d4617a7f2f0d1: Status 404 returned error can't find the container with id 3b563c0379586a7343c34300d5e9a6b9e17b1262e823e8a6d23d4617a7f2f0d1
Aug 04 18:21:05.833352 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:21:05.833257    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f9jftff" event={"ID":"2256be5b-f6a8-4be7-b774-27c21ddfa89f","Type":"ContainerStarted","Data":"c233698fae967dc2abdc75943a99ecaaaed4872abc6ec146798675353b363336"}
Aug 04 18:21:05.833352 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:21:05.833310    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f9jftff" event={"ID":"2256be5b-f6a8-4be7-b774-27c21ddfa89f","Type":"ContainerStarted","Data":"3b563c0379586a7343c34300d5e9a6b9e17b1262e823e8a6d23d4617a7f2f0d1"}
Aug 04 18:21:05.833604 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:21:05.833370    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f9jftff"
Aug 04 18:21:05.850917 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:21:05.850869    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f9jftff" podStartSLOduration=0.850854162 podStartE2EDuration="850.854162ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:21:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:21:05.849643752 +0000 UTC m=+1279.430862332" watchObservedRunningTime="2026-08-04 18:21:05.850854162 +0000 UTC m=+1279.432072740"
Aug 04 18:21:20.302287 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:21:20.302242    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-fcb47dfd4-mthzq"]
Aug 04 18:21:20.306132 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:21:20.306109    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-fcb47dfd4-mthzq"
Aug 04 18:21:20.309080 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:21:20.309052    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"openshift-service-ca.crt\""
Aug 04 18:21:20.309205 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:21:20.309057    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"seaweedfs-s3-creds\""
Aug 04 18:21:20.309205 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:21:20.309142    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"odh-kserve-custom-ca-bundle\""
Aug 04 18:21:20.309205 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:21:20.309163    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"kube-root-ca.crt\""
Aug 04 18:21:20.309388 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:21:20.309248    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"auth-disabled-test-kserve-self-signed-certs\""
Aug 04 18:21:20.309455 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:21:20.309444    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"default-dockercfg-d9wnb\""
Aug 04 18:21:20.317759 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:21:20.317737    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-fcb47dfd4-mthzq"]
Aug 04 18:21:20.360981 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:21:20.360952    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fba83362-7bb6-4c74-b0e5-1e9cf25ac79e-tmp-dir\") pod \"auth-disabled-test-kserve-fcb47dfd4-mthzq\" (UID: \"fba83362-7bb6-4c74-b0e5-1e9cf25ac79e\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-fcb47dfd4-mthzq"
Aug 04 18:21:20.361108 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:21:20.360984    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fba83362-7bb6-4c74-b0e5-1e9cf25ac79e-model-cache\") pod \"auth-disabled-test-kserve-fcb47dfd4-mthzq\" (UID: \"fba83362-7bb6-4c74-b0e5-1e9cf25ac79e\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-fcb47dfd4-mthzq"
Aug 04 18:21:20.361108 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:21:20.361007    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fba83362-7bb6-4c74-b0e5-1e9cf25ac79e-home\") pod \"auth-disabled-test-kserve-fcb47dfd4-mthzq\" (UID: \"fba83362-7bb6-4c74-b0e5-1e9cf25ac79e\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-fcb47dfd4-mthzq"
Aug 04 18:21:20.361108 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:21:20.361086    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fba83362-7bb6-4c74-b0e5-1e9cf25ac79e-dshm\") pod \"auth-disabled-test-kserve-fcb47dfd4-mthzq\" (UID: \"fba83362-7bb6-4c74-b0e5-1e9cf25ac79e\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-fcb47dfd4-mthzq"
Aug 04 18:21:20.361217 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:21:20.361177    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fba83362-7bb6-4c74-b0e5-1e9cf25ac79e-kserve-provision-location\") pod \"auth-disabled-test-kserve-fcb47dfd4-mthzq\" (UID: \"fba83362-7bb6-4c74-b0e5-1e9cf25ac79e\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-fcb47dfd4-mthzq"
Aug 04 18:21:20.361217 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:21:20.361199    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fba83362-7bb6-4c74-b0e5-1e9cf25ac79e-cabundle-cert\") pod \"auth-disabled-test-kserve-fcb47dfd4-mthzq\" (UID: \"fba83362-7bb6-4c74-b0e5-1e9cf25ac79e\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-fcb47dfd4-mthzq"
Aug 04 18:21:20.361285 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:21:20.361234    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6dw2\" (UniqueName: \"kubernetes.io/projected/fba83362-7bb6-4c74-b0e5-1e9cf25ac79e-kube-api-access-n6dw2\") pod \"auth-disabled-test-kserve-fcb47dfd4-mthzq\" (UID: \"fba83362-7bb6-4c74-b0e5-1e9cf25ac79e\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-fcb47dfd4-mthzq"
Aug 04 18:21:20.361285 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:21:20.361258    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fba83362-7bb6-4c74-b0e5-1e9cf25ac79e-tls-certs\") pod \"auth-disabled-test-kserve-fcb47dfd4-mthzq\" (UID: \"fba83362-7bb6-4c74-b0e5-1e9cf25ac79e\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-fcb47dfd4-mthzq"
Aug 04 18:21:20.462258 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:21:20.462228    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fba83362-7bb6-4c74-b0e5-1e9cf25ac79e-kserve-provision-location\") pod \"auth-disabled-test-kserve-fcb47dfd4-mthzq\" (UID: \"fba83362-7bb6-4c74-b0e5-1e9cf25ac79e\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-fcb47dfd4-mthzq"
Aug 04 18:21:20.462402 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:21:20.462261    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fba83362-7bb6-4c74-b0e5-1e9cf25ac79e-cabundle-cert\") pod \"auth-disabled-test-kserve-fcb47dfd4-mthzq\" (UID: \"fba83362-7bb6-4c74-b0e5-1e9cf25ac79e\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-fcb47dfd4-mthzq"
Aug 04 18:21:20.462402 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:21:20.462282    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6dw2\" (UniqueName: \"kubernetes.io/projected/fba83362-7bb6-4c74-b0e5-1e9cf25ac79e-kube-api-access-n6dw2\") pod \"auth-disabled-test-kserve-fcb47dfd4-mthzq\" (UID: \"fba83362-7bb6-4c74-b0e5-1e9cf25ac79e\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-fcb47dfd4-mthzq"
Aug 04 18:21:20.462402 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:21:20.462315    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fba83362-7bb6-4c74-b0e5-1e9cf25ac79e-tls-certs\") pod \"auth-disabled-test-kserve-fcb47dfd4-mthzq\" (UID: \"fba83362-7bb6-4c74-b0e5-1e9cf25ac79e\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-fcb47dfd4-mthzq"
Aug 04 18:21:20.462402 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:21:20.462349    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fba83362-7bb6-4c74-b0e5-1e9cf25ac79e-tmp-dir\") pod \"auth-disabled-test-kserve-fcb47dfd4-mthzq\" (UID: \"fba83362-7bb6-4c74-b0e5-1e9cf25ac79e\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-fcb47dfd4-mthzq"
Aug 04 18:21:20.462402 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:21:20.462374    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fba83362-7bb6-4c74-b0e5-1e9cf25ac79e-model-cache\") pod \"auth-disabled-test-kserve-fcb47dfd4-mthzq\" (UID: \"fba83362-7bb6-4c74-b0e5-1e9cf25ac79e\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-fcb47dfd4-mthzq"
Aug 04 18:21:20.462402 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:21:20.462402    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fba83362-7bb6-4c74-b0e5-1e9cf25ac79e-home\") pod \"auth-disabled-test-kserve-fcb47dfd4-mthzq\" (UID: \"fba83362-7bb6-4c74-b0e5-1e9cf25ac79e\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-fcb47dfd4-mthzq"
Aug 04 18:21:20.462734 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:21:20.462442    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fba83362-7bb6-4c74-b0e5-1e9cf25ac79e-dshm\") pod \"auth-disabled-test-kserve-fcb47dfd4-mthzq\" (UID: \"fba83362-7bb6-4c74-b0e5-1e9cf25ac79e\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-fcb47dfd4-mthzq"
Aug 04 18:21:20.462734 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:21:20.462708    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fba83362-7bb6-4c74-b0e5-1e9cf25ac79e-kserve-provision-location\") pod \"auth-disabled-test-kserve-fcb47dfd4-mthzq\" (UID: \"fba83362-7bb6-4c74-b0e5-1e9cf25ac79e\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-fcb47dfd4-mthzq"
Aug 04 18:21:20.462840 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:21:20.462772    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fba83362-7bb6-4c74-b0e5-1e9cf25ac79e-model-cache\") pod \"auth-disabled-test-kserve-fcb47dfd4-mthzq\" (UID: \"fba83362-7bb6-4c74-b0e5-1e9cf25ac79e\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-fcb47dfd4-mthzq"
Aug 04 18:21:20.462840 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:21:20.462812    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fba83362-7bb6-4c74-b0e5-1e9cf25ac79e-tmp-dir\") pod \"auth-disabled-test-kserve-fcb47dfd4-mthzq\" (UID: \"fba83362-7bb6-4c74-b0e5-1e9cf25ac79e\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-fcb47dfd4-mthzq"
Aug 04 18:21:20.462951 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:21:20.462858    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fba83362-7bb6-4c74-b0e5-1e9cf25ac79e-home\") pod \"auth-disabled-test-kserve-fcb47dfd4-mthzq\" (UID: \"fba83362-7bb6-4c74-b0e5-1e9cf25ac79e\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-fcb47dfd4-mthzq"
Aug 04 18:21:20.463263 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:21:20.463230    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fba83362-7bb6-4c74-b0e5-1e9cf25ac79e-cabundle-cert\") pod \"auth-disabled-test-kserve-fcb47dfd4-mthzq\" (UID: \"fba83362-7bb6-4c74-b0e5-1e9cf25ac79e\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-fcb47dfd4-mthzq"
Aug 04 18:21:20.464821 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:21:20.464799    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fba83362-7bb6-4c74-b0e5-1e9cf25ac79e-dshm\") pod \"auth-disabled-test-kserve-fcb47dfd4-mthzq\" (UID: \"fba83362-7bb6-4c74-b0e5-1e9cf25ac79e\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-fcb47dfd4-mthzq"
Aug 04 18:21:20.465299 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:21:20.465283    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fba83362-7bb6-4c74-b0e5-1e9cf25ac79e-tls-certs\") pod \"auth-disabled-test-kserve-fcb47dfd4-mthzq\" (UID: \"fba83362-7bb6-4c74-b0e5-1e9cf25ac79e\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-fcb47dfd4-mthzq"
Aug 04 18:21:20.471209 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:21:20.471187    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6dw2\" (UniqueName: \"kubernetes.io/projected/fba83362-7bb6-4c74-b0e5-1e9cf25ac79e-kube-api-access-n6dw2\") pod \"auth-disabled-test-kserve-fcb47dfd4-mthzq\" (UID: \"fba83362-7bb6-4c74-b0e5-1e9cf25ac79e\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-fcb47dfd4-mthzq"
Aug 04 18:21:20.619023 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:21:20.618926    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-fcb47dfd4-mthzq"
Aug 04 18:21:20.762235 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:21:20.762204    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-fcb47dfd4-mthzq"]
Aug 04 18:21:20.765518 ip-10-0-135-124 kubenswrapper[2583]: W0804 18:21:20.765486    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfba83362_7bb6_4c74_b0e5_1e9cf25ac79e.slice/crio-689cb9470b7235b940a81ef7c4e32e65f920b547a77766bc07afb59b568a1fb1 WatchSource:0}: Error finding container 689cb9470b7235b940a81ef7c4e32e65f920b547a77766bc07afb59b568a1fb1: Status 404 returned error can't find the container with id 689cb9470b7235b940a81ef7c4e32e65f920b547a77766bc07afb59b568a1fb1
Aug 04 18:21:20.914586 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:21:20.914490    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-fcb47dfd4-mthzq" event={"ID":"fba83362-7bb6-4c74-b0e5-1e9cf25ac79e","Type":"ContainerStarted","Data":"736c091ad3e5e7f80803eb12116b387475b24a6b108d06a31a595ef03f1ee1de"}
Aug 04 18:21:20.914586 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:21:20.914556    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-fcb47dfd4-mthzq" event={"ID":"fba83362-7bb6-4c74-b0e5-1e9cf25ac79e","Type":"ContainerStarted","Data":"689cb9470b7235b940a81ef7c4e32e65f920b547a77766bc07afb59b568a1fb1"}
Aug 04 18:21:23.931919 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:21:23.931886    2583 generic.go:356] "Generic (PLEG): container finished" podID="fba83362-7bb6-4c74-b0e5-1e9cf25ac79e" containerID="736c091ad3e5e7f80803eb12116b387475b24a6b108d06a31a595ef03f1ee1de" exitCode=0
Aug 04 18:21:23.932420 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:21:23.931959    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-fcb47dfd4-mthzq" event={"ID":"fba83362-7bb6-4c74-b0e5-1e9cf25ac79e","Type":"ContainerDied","Data":"736c091ad3e5e7f80803eb12116b387475b24a6b108d06a31a595ef03f1ee1de"}
Aug 04 18:21:24.942138 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:21:24.942097    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-fcb47dfd4-mthzq" event={"ID":"fba83362-7bb6-4c74-b0e5-1e9cf25ac79e","Type":"ContainerStarted","Data":"1863a679cdd81526be65ebd892e7a5282712fe1d2d253a54e00c87e731dac023"}
Aug 04 18:21:24.968244 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:21:24.968194    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-fcb47dfd4-mthzq" podStartSLOduration=2.859597016 podStartE2EDuration="4.968179313s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.108582297s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:21:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:21:24.965567196 +0000 UTC m=+1298.546785774" watchObservedRunningTime="2026-08-04 18:21:24.968179313 +0000 UTC m=+1298.549397891"
Aug 04 18:21:25.619028 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:21:25.618996    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-fcb47dfd4-mthzq"
Aug 04 18:21:30.619835 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:21:30.619791    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-fcb47dfd4-mthzq"
Aug 04 18:21:30.621636 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:21:30.621606    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-fcb47dfd4-mthzq" podUID="fba83362-7bb6-4c74-b0e5-1e9cf25ac79e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/health\": dial tcp 10.132.0.56:8000: connect: connection refused"
Aug 04 18:21:36.841227 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:21:36.841197    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f9jftff"
Aug 04 18:21:40.619890 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:21:40.619844    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-fcb47dfd4-mthzq" podUID="fba83362-7bb6-4c74-b0e5-1e9cf25ac79e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/health\": dial tcp 10.132.0.56:8000: connect: connection refused"
Aug 04 18:21:50.620192 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:21:50.620152    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-fcb47dfd4-mthzq" podUID="fba83362-7bb6-4c74-b0e5-1e9cf25ac79e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/health\": dial tcp 10.132.0.56:8000: connect: connection refused"
Aug 04 18:22:00.619382 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:22:00.619335    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-fcb47dfd4-mthzq" podUID="fba83362-7bb6-4c74-b0e5-1e9cf25ac79e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/health\": dial tcp 10.132.0.56:8000: connect: connection refused"
Aug 04 18:22:10.619828 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:22:10.619734    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-fcb47dfd4-mthzq" podUID="fba83362-7bb6-4c74-b0e5-1e9cf25ac79e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/health\": dial tcp 10.132.0.56:8000: connect: connection refused"
Aug 04 18:22:20.620007 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:22:20.619967    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-fcb47dfd4-mthzq" podUID="fba83362-7bb6-4c74-b0e5-1e9cf25ac79e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/health\": dial tcp 10.132.0.56:8000: connect: connection refused"
Aug 04 18:22:30.620275 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:22:30.620231    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-fcb47dfd4-mthzq" podUID="fba83362-7bb6-4c74-b0e5-1e9cf25ac79e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/health\": dial tcp 10.132.0.56:8000: connect: connection refused"
Aug 04 18:22:40.620169 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:22:40.620124    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-fcb47dfd4-mthzq" podUID="fba83362-7bb6-4c74-b0e5-1e9cf25ac79e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/health\": dial tcp 10.132.0.56:8000: connect: connection refused"
Aug 04 18:22:43.167931 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:22:43.167893    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f9jftff"]
Aug 04 18:22:46.840876 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:22:46.840840    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f9jftff" podUID="2256be5b-f6a8-4be7-b774-27c21ddfa89f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Aug 04 18:22:50.619636 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:22:50.619581    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-fcb47dfd4-mthzq" podUID="fba83362-7bb6-4c74-b0e5-1e9cf25ac79e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/health\": dial tcp 10.132.0.56:8000: connect: connection refused"
Aug 04 18:22:56.841451 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:22:56.841410    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f9jftff" podUID="2256be5b-f6a8-4be7-b774-27c21ddfa89f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Aug 04 18:22:58.187942 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:22:58.187872    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f9jftff" podUID="2256be5b-f6a8-4be7-b774-27c21ddfa89f" containerName="main" containerID="cri-o://c233698fae967dc2abdc75943a99ecaaaed4872abc6ec146798675353b363336" gracePeriod=45
Aug 04 18:22:58.444276 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:22:58.444204    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f9jftff"
Aug 04 18:22:58.484175 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:22:58.484137    2583 generic.go:356] "Generic (PLEG): container finished" podID="2256be5b-f6a8-4be7-b774-27c21ddfa89f" containerID="c233698fae967dc2abdc75943a99ecaaaed4872abc6ec146798675353b363336" exitCode=0
Aug 04 18:22:58.484175 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:22:58.484183    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f9jftff"
Aug 04 18:22:58.484395 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:22:58.484216    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f9jftff" event={"ID":"2256be5b-f6a8-4be7-b774-27c21ddfa89f","Type":"ContainerDied","Data":"c233698fae967dc2abdc75943a99ecaaaed4872abc6ec146798675353b363336"}
Aug 04 18:22:58.484395 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:22:58.484304    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f9jftff" event={"ID":"2256be5b-f6a8-4be7-b774-27c21ddfa89f","Type":"ContainerDied","Data":"3b563c0379586a7343c34300d5e9a6b9e17b1262e823e8a6d23d4617a7f2f0d1"}
Aug 04 18:22:58.484395 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:22:58.484330    2583 scope.go:117] "RemoveContainer" containerID="c233698fae967dc2abdc75943a99ecaaaed4872abc6ec146798675353b363336"
Aug 04 18:22:58.493232 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:22:58.493211    2583 scope.go:117] "RemoveContainer" containerID="c233698fae967dc2abdc75943a99ecaaaed4872abc6ec146798675353b363336"
Aug 04 18:22:58.493518 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:22:58.493495    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c233698fae967dc2abdc75943a99ecaaaed4872abc6ec146798675353b363336\": container with ID starting with c233698fae967dc2abdc75943a99ecaaaed4872abc6ec146798675353b363336 not found: ID does not exist" containerID="c233698fae967dc2abdc75943a99ecaaaed4872abc6ec146798675353b363336"
Aug 04 18:22:58.493601 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:22:58.493531    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c233698fae967dc2abdc75943a99ecaaaed4872abc6ec146798675353b363336"} err="failed to get container status \"c233698fae967dc2abdc75943a99ecaaaed4872abc6ec146798675353b363336\": rpc error: code = NotFound desc = could not find container \"c233698fae967dc2abdc75943a99ecaaaed4872abc6ec146798675353b363336\": container with ID starting with c233698fae967dc2abdc75943a99ecaaaed4872abc6ec146798675353b363336 not found: ID does not exist"
Aug 04 18:22:58.634994 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:22:58.634953    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5xhlj\" (UniqueName: \"kubernetes.io/projected/2256be5b-f6a8-4be7-b774-27c21ddfa89f-kube-api-access-5xhlj\") pod \"2256be5b-f6a8-4be7-b774-27c21ddfa89f\" (UID: \"2256be5b-f6a8-4be7-b774-27c21ddfa89f\") "
Aug 04 18:22:58.635164 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:22:58.635064    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2256be5b-f6a8-4be7-b774-27c21ddfa89f-tls-certs\") pod \"2256be5b-f6a8-4be7-b774-27c21ddfa89f\" (UID: \"2256be5b-f6a8-4be7-b774-27c21ddfa89f\") "
Aug 04 18:22:58.637257 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:22:58.637224    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2256be5b-f6a8-4be7-b774-27c21ddfa89f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2256be5b-f6a8-4be7-b774-27c21ddfa89f" (UID: "2256be5b-f6a8-4be7-b774-27c21ddfa89f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:22:58.637382 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:22:58.637234    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2256be5b-f6a8-4be7-b774-27c21ddfa89f-kube-api-access-5xhlj" (OuterVolumeSpecName: "kube-api-access-5xhlj") pod "2256be5b-f6a8-4be7-b774-27c21ddfa89f" (UID: "2256be5b-f6a8-4be7-b774-27c21ddfa89f"). InnerVolumeSpecName "kube-api-access-5xhlj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:22:58.736172 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:22:58.736073    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5xhlj\" (UniqueName: \"kubernetes.io/projected/2256be5b-f6a8-4be7-b774-27c21ddfa89f-kube-api-access-5xhlj\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:22:58.736172 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:22:58.736126    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2256be5b-f6a8-4be7-b774-27c21ddfa89f-tls-certs\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:22:58.817405 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:22:58.817371    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f9jftff"]
Aug 04 18:22:58.823822 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:22:58.823786    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f9jftff"]
Aug 04 18:22:58.944545 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:22:58.944509    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2256be5b-f6a8-4be7-b774-27c21ddfa89f" path="/var/lib/kubelet/pods/2256be5b-f6a8-4be7-b774-27c21ddfa89f/volumes"
Aug 04 18:23:00.620403 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:23:00.620358    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-fcb47dfd4-mthzq" podUID="fba83362-7bb6-4c74-b0e5-1e9cf25ac79e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/health\": dial tcp 10.132.0.56:8000: connect: connection refused"
Aug 04 18:23:10.619907 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:23:10.619865    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-fcb47dfd4-mthzq" podUID="fba83362-7bb6-4c74-b0e5-1e9cf25ac79e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/health\": dial tcp 10.132.0.56:8000: connect: connection refused"
Aug 04 18:23:17.676025 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:23:17.675990    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedbzrq7"]
Aug 04 18:23:17.677111 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:23:17.676684    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2256be5b-f6a8-4be7-b774-27c21ddfa89f" containerName="main"
Aug 04 18:23:17.677111 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:23:17.676705    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="2256be5b-f6a8-4be7-b774-27c21ddfa89f" containerName="main"
Aug 04 18:23:17.677111 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:23:17.676831    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="2256be5b-f6a8-4be7-b774-27c21ddfa89f" containerName="main"
Aug 04 18:23:17.679884 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:23:17.679862    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedbzrq7"
Aug 04 18:23:17.683981 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:23:17.683957    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"openshift-service-ca.crt\""
Aug 04 18:23:17.683981 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:23:17.683970    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"scheduler-v06-threshold-migration-test-epp-sa-dockercfg-l94p7\""
Aug 04 18:23:17.684146 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:23:17.683959    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"default-dockercfg-jczgf\""
Aug 04 18:23:17.684146 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:23:17.684013    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"kube-root-ca.crt\""
Aug 04 18:23:17.684146 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:23:17.684019    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"scheduler-v06-threshold-migration-test-kserve-self-signed-certs\""
Aug 04 18:23:17.692112 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:23:17.692092    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedbzrq7"]
Aug 04 18:23:17.802679 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:23:17.802640    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d840eb0-a195-4e73-969a-0a478f5b16fd-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedbzrq7\" (UID: \"2d840eb0-a195-4e73-969a-0a478f5b16fd\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedbzrq7"
Aug 04 18:23:17.802845 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:23:17.802685    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5sxrr\" (UniqueName: \"kubernetes.io/projected/2d840eb0-a195-4e73-969a-0a478f5b16fd-kube-api-access-5sxrr\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedbzrq7\" (UID: \"2d840eb0-a195-4e73-969a-0a478f5b16fd\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedbzrq7"
Aug 04 18:23:17.903671 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:23:17.903631    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d840eb0-a195-4e73-969a-0a478f5b16fd-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedbzrq7\" (UID: \"2d840eb0-a195-4e73-969a-0a478f5b16fd\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedbzrq7"
Aug 04 18:23:17.903842 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:23:17.903678    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5sxrr\" (UniqueName: \"kubernetes.io/projected/2d840eb0-a195-4e73-969a-0a478f5b16fd-kube-api-access-5sxrr\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedbzrq7\" (UID: \"2d840eb0-a195-4e73-969a-0a478f5b16fd\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedbzrq7"
Aug 04 18:23:17.906450 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:23:17.906421    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d840eb0-a195-4e73-969a-0a478f5b16fd-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedbzrq7\" (UID: \"2d840eb0-a195-4e73-969a-0a478f5b16fd\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedbzrq7"
Aug 04 18:23:17.912399 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:23:17.912375    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5sxrr\" (UniqueName: \"kubernetes.io/projected/2d840eb0-a195-4e73-969a-0a478f5b16fd-kube-api-access-5sxrr\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedbzrq7\" (UID: \"2d840eb0-a195-4e73-969a-0a478f5b16fd\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedbzrq7"
Aug 04 18:23:17.990558 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:23:17.990522    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedbzrq7"
Aug 04 18:23:18.127767 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:23:18.127727    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedbzrq7"]
Aug 04 18:23:18.128968 ip-10-0-135-124 kubenswrapper[2583]: W0804 18:23:18.128939    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d840eb0_a195_4e73_969a_0a478f5b16fd.slice/crio-ee18b4e8ff5c7e3c57a608bf1d296f675012b34ff706e8ced058e659ac5b59ea WatchSource:0}: Error finding container ee18b4e8ff5c7e3c57a608bf1d296f675012b34ff706e8ced058e659ac5b59ea: Status 404 returned error can't find the container with id ee18b4e8ff5c7e3c57a608bf1d296f675012b34ff706e8ced058e659ac5b59ea
Aug 04 18:23:18.602456 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:23:18.602408    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedbzrq7" event={"ID":"2d840eb0-a195-4e73-969a-0a478f5b16fd","Type":"ContainerStarted","Data":"4e7843e01b86192aab3fcb5974bfdc3f20cd7768d6f8d6a0a9e5320a27fbb385"}
Aug 04 18:23:18.602668 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:23:18.602496    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedbzrq7" event={"ID":"2d840eb0-a195-4e73-969a-0a478f5b16fd","Type":"ContainerStarted","Data":"ee18b4e8ff5c7e3c57a608bf1d296f675012b34ff706e8ced058e659ac5b59ea"}
Aug 04 18:23:18.602668 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:23:18.602525    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedbzrq7"
Aug 04 18:23:18.626026 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:23:18.625974    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedbzrq7" podStartSLOduration=1.625959193 podStartE2EDuration="1.625959193s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:23:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:23:18.62479732 +0000 UTC m=+1412.206015898" watchObservedRunningTime="2026-08-04 18:23:18.625959193 +0000 UTC m=+1412.207177773"
Aug 04 18:23:20.629567 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:23:20.629521    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-fcb47dfd4-mthzq"
Aug 04 18:23:20.637797 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:23:20.637768    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-fcb47dfd4-mthzq"
Aug 04 18:23:38.216628 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:23:38.216589    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-fcb47dfd4-mthzq"]
Aug 04 18:23:49.610304 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:23:49.610272    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedbzrq7"
Aug 04 18:23:51.750580 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:23:51.750499    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedbzrq7"]
Aug 04 18:23:53.239840 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:23:53.239802    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-fcb47dfd4-mthzq" podUID="fba83362-7bb6-4c74-b0e5-1e9cf25ac79e" containerName="main" containerID="cri-o://1863a679cdd81526be65ebd892e7a5282712fe1d2d253a54e00c87e731dac023" gracePeriod=45
Aug 04 18:23:53.630633 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:23:53.630538    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-fcb47dfd4-mthzq" podUID="fba83362-7bb6-4c74-b0e5-1e9cf25ac79e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/health\": dial tcp 10.132.0.56:8000: connect: connection refused"
Aug 04 18:23:53.988973 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:23:53.988950    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-fcb47dfd4-mthzq"
Aug 04 18:23:54.127494 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:23:54.127431    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fba83362-7bb6-4c74-b0e5-1e9cf25ac79e-tmp-dir\") pod \"fba83362-7bb6-4c74-b0e5-1e9cf25ac79e\" (UID: \"fba83362-7bb6-4c74-b0e5-1e9cf25ac79e\") "
Aug 04 18:23:54.127696 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:23:54.127518    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fba83362-7bb6-4c74-b0e5-1e9cf25ac79e-home\") pod \"fba83362-7bb6-4c74-b0e5-1e9cf25ac79e\" (UID: \"fba83362-7bb6-4c74-b0e5-1e9cf25ac79e\") "
Aug 04 18:23:54.127696 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:23:54.127538    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n6dw2\" (UniqueName: \"kubernetes.io/projected/fba83362-7bb6-4c74-b0e5-1e9cf25ac79e-kube-api-access-n6dw2\") pod \"fba83362-7bb6-4c74-b0e5-1e9cf25ac79e\" (UID: \"fba83362-7bb6-4c74-b0e5-1e9cf25ac79e\") "
Aug 04 18:23:54.127696 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:23:54.127566    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fba83362-7bb6-4c74-b0e5-1e9cf25ac79e-cabundle-cert\") pod \"fba83362-7bb6-4c74-b0e5-1e9cf25ac79e\" (UID: \"fba83362-7bb6-4c74-b0e5-1e9cf25ac79e\") "
Aug 04 18:23:54.127696 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:23:54.127608    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fba83362-7bb6-4c74-b0e5-1e9cf25ac79e-model-cache\") pod \"fba83362-7bb6-4c74-b0e5-1e9cf25ac79e\" (UID: \"fba83362-7bb6-4c74-b0e5-1e9cf25ac79e\") "
Aug 04 18:23:54.127696 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:23:54.127666    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fba83362-7bb6-4c74-b0e5-1e9cf25ac79e-tls-certs\") pod \"fba83362-7bb6-4c74-b0e5-1e9cf25ac79e\" (UID: \"fba83362-7bb6-4c74-b0e5-1e9cf25ac79e\") "
Aug 04 18:23:54.127984 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:23:54.127735    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fba83362-7bb6-4c74-b0e5-1e9cf25ac79e-kserve-provision-location\") pod \"fba83362-7bb6-4c74-b0e5-1e9cf25ac79e\" (UID: \"fba83362-7bb6-4c74-b0e5-1e9cf25ac79e\") "
Aug 04 18:23:54.127984 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:23:54.127784    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fba83362-7bb6-4c74-b0e5-1e9cf25ac79e-dshm\") pod \"fba83362-7bb6-4c74-b0e5-1e9cf25ac79e\" (UID: \"fba83362-7bb6-4c74-b0e5-1e9cf25ac79e\") "
Aug 04 18:23:54.127984 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:23:54.127848    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fba83362-7bb6-4c74-b0e5-1e9cf25ac79e-model-cache" (OuterVolumeSpecName: "model-cache") pod "fba83362-7bb6-4c74-b0e5-1e9cf25ac79e" (UID: "fba83362-7bb6-4c74-b0e5-1e9cf25ac79e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:23:54.127984 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:23:54.127968    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fba83362-7bb6-4c74-b0e5-1e9cf25ac79e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "fba83362-7bb6-4c74-b0e5-1e9cf25ac79e" (UID: "fba83362-7bb6-4c74-b0e5-1e9cf25ac79e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:23:54.128177 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:23:54.128109    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fba83362-7bb6-4c74-b0e5-1e9cf25ac79e-cabundle-cert\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:23:54.128177 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:23:54.128128    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fba83362-7bb6-4c74-b0e5-1e9cf25ac79e-model-cache\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:23:54.128285 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:23:54.128230    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fba83362-7bb6-4c74-b0e5-1e9cf25ac79e-home" (OuterVolumeSpecName: "home") pod "fba83362-7bb6-4c74-b0e5-1e9cf25ac79e" (UID: "fba83362-7bb6-4c74-b0e5-1e9cf25ac79e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:23:54.130011 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:23:54.129977    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fba83362-7bb6-4c74-b0e5-1e9cf25ac79e-kube-api-access-n6dw2" (OuterVolumeSpecName: "kube-api-access-n6dw2") pod "fba83362-7bb6-4c74-b0e5-1e9cf25ac79e" (UID: "fba83362-7bb6-4c74-b0e5-1e9cf25ac79e"). InnerVolumeSpecName "kube-api-access-n6dw2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:23:54.130140 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:23:54.130068    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fba83362-7bb6-4c74-b0e5-1e9cf25ac79e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fba83362-7bb6-4c74-b0e5-1e9cf25ac79e" (UID: "fba83362-7bb6-4c74-b0e5-1e9cf25ac79e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:23:54.130283 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:23:54.130260    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fba83362-7bb6-4c74-b0e5-1e9cf25ac79e-dshm" (OuterVolumeSpecName: "dshm") pod "fba83362-7bb6-4c74-b0e5-1e9cf25ac79e" (UID: "fba83362-7bb6-4c74-b0e5-1e9cf25ac79e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:23:54.141366 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:23:54.141338    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fba83362-7bb6-4c74-b0e5-1e9cf25ac79e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "fba83362-7bb6-4c74-b0e5-1e9cf25ac79e" (UID: "fba83362-7bb6-4c74-b0e5-1e9cf25ac79e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:23:54.150672 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:23:54.150646    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fba83362-7bb6-4c74-b0e5-1e9cf25ac79e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "fba83362-7bb6-4c74-b0e5-1e9cf25ac79e" (UID: "fba83362-7bb6-4c74-b0e5-1e9cf25ac79e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:23:54.228729 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:23:54.228698    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fba83362-7bb6-4c74-b0e5-1e9cf25ac79e-tls-certs\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:23:54.228729 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:23:54.228725    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fba83362-7bb6-4c74-b0e5-1e9cf25ac79e-kserve-provision-location\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:23:54.228944 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:23:54.228735    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fba83362-7bb6-4c74-b0e5-1e9cf25ac79e-dshm\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:23:54.228944 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:23:54.228744    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fba83362-7bb6-4c74-b0e5-1e9cf25ac79e-tmp-dir\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:23:54.228944 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:23:54.228753    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fba83362-7bb6-4c74-b0e5-1e9cf25ac79e-home\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:23:54.228944 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:23:54.228761    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n6dw2\" (UniqueName: \"kubernetes.io/projected/fba83362-7bb6-4c74-b0e5-1e9cf25ac79e-kube-api-access-n6dw2\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:23:54.811353 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:23:54.811319    2583 generic.go:356] "Generic (PLEG): container finished" podID="fba83362-7bb6-4c74-b0e5-1e9cf25ac79e" containerID="1863a679cdd81526be65ebd892e7a5282712fe1d2d253a54e00c87e731dac023" exitCode=0
Aug 04 18:23:54.811846 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:23:54.811400    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-fcb47dfd4-mthzq"
Aug 04 18:23:54.811846 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:23:54.811408    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-fcb47dfd4-mthzq" event={"ID":"fba83362-7bb6-4c74-b0e5-1e9cf25ac79e","Type":"ContainerDied","Data":"1863a679cdd81526be65ebd892e7a5282712fe1d2d253a54e00c87e731dac023"}
Aug 04 18:23:54.811846 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:23:54.811567    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-fcb47dfd4-mthzq" event={"ID":"fba83362-7bb6-4c74-b0e5-1e9cf25ac79e","Type":"ContainerDied","Data":"689cb9470b7235b940a81ef7c4e32e65f920b547a77766bc07afb59b568a1fb1"}
Aug 04 18:23:54.811846 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:23:54.811603    2583 scope.go:117] "RemoveContainer" containerID="1863a679cdd81526be65ebd892e7a5282712fe1d2d253a54e00c87e731dac023"
Aug 04 18:23:54.820851 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:23:54.820836    2583 scope.go:117] "RemoveContainer" containerID="736c091ad3e5e7f80803eb12116b387475b24a6b108d06a31a595ef03f1ee1de"
Aug 04 18:23:54.828876 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:23:54.828857    2583 scope.go:117] "RemoveContainer" containerID="1863a679cdd81526be65ebd892e7a5282712fe1d2d253a54e00c87e731dac023"
Aug 04 18:23:54.829124 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:23:54.829106    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1863a679cdd81526be65ebd892e7a5282712fe1d2d253a54e00c87e731dac023\": container with ID starting with 1863a679cdd81526be65ebd892e7a5282712fe1d2d253a54e00c87e731dac023 not found: ID does not exist" containerID="1863a679cdd81526be65ebd892e7a5282712fe1d2d253a54e00c87e731dac023"
Aug 04 18:23:54.829171 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:23:54.829132    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1863a679cdd81526be65ebd892e7a5282712fe1d2d253a54e00c87e731dac023"} err="failed to get container status \"1863a679cdd81526be65ebd892e7a5282712fe1d2d253a54e00c87e731dac023\": rpc error: code = NotFound desc = could not find container \"1863a679cdd81526be65ebd892e7a5282712fe1d2d253a54e00c87e731dac023\": container with ID starting with 1863a679cdd81526be65ebd892e7a5282712fe1d2d253a54e00c87e731dac023 not found: ID does not exist"
Aug 04 18:23:54.829171 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:23:54.829148    2583 scope.go:117] "RemoveContainer" containerID="736c091ad3e5e7f80803eb12116b387475b24a6b108d06a31a595ef03f1ee1de"
Aug 04 18:23:54.829348 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:23:54.829335    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"736c091ad3e5e7f80803eb12116b387475b24a6b108d06a31a595ef03f1ee1de\": container with ID starting with 736c091ad3e5e7f80803eb12116b387475b24a6b108d06a31a595ef03f1ee1de not found: ID does not exist" containerID="736c091ad3e5e7f80803eb12116b387475b24a6b108d06a31a595ef03f1ee1de"
Aug 04 18:23:54.829397 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:23:54.829351    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"736c091ad3e5e7f80803eb12116b387475b24a6b108d06a31a595ef03f1ee1de"} err="failed to get container status \"736c091ad3e5e7f80803eb12116b387475b24a6b108d06a31a595ef03f1ee1de\": rpc error: code = NotFound desc = could not find container \"736c091ad3e5e7f80803eb12116b387475b24a6b108d06a31a595ef03f1ee1de\": container with ID starting with 736c091ad3e5e7f80803eb12116b387475b24a6b108d06a31a595ef03f1ee1de not found: ID does not exist"
Aug 04 18:23:54.837653 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:23:54.837633    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-fcb47dfd4-mthzq"]
Aug 04 18:23:54.842756 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:23:54.842736    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-fcb47dfd4-mthzq"]
Aug 04 18:23:54.941727 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:23:54.941703    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fba83362-7bb6-4c74-b0e5-1e9cf25ac79e" path="/var/lib/kubelet/pods/fba83362-7bb6-4c74-b0e5-1e9cf25ac79e/volumes"
Aug 04 18:23:59.609814 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:23:59.609770    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedbzrq7" podUID="2d840eb0-a195-4e73-969a-0a478f5b16fd" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Aug 04 18:24:06.770970 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:24:06.770925    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedbzrq7" podUID="2d840eb0-a195-4e73-969a-0a478f5b16fd" containerName="main" containerID="cri-o://4e7843e01b86192aab3fcb5974bfdc3f20cd7768d6f8d6a0a9e5320a27fbb385" gracePeriod=45
Aug 04 18:24:07.030703 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:24:07.030643    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedbzrq7"
Aug 04 18:24:07.119717 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:24:07.119687    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5sxrr\" (UniqueName: \"kubernetes.io/projected/2d840eb0-a195-4e73-969a-0a478f5b16fd-kube-api-access-5sxrr\") pod \"2d840eb0-a195-4e73-969a-0a478f5b16fd\" (UID: \"2d840eb0-a195-4e73-969a-0a478f5b16fd\") "
Aug 04 18:24:07.121879 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:24:07.121855    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d840eb0-a195-4e73-969a-0a478f5b16fd-kube-api-access-5sxrr" (OuterVolumeSpecName: "kube-api-access-5sxrr") pod "2d840eb0-a195-4e73-969a-0a478f5b16fd" (UID: "2d840eb0-a195-4e73-969a-0a478f5b16fd"). InnerVolumeSpecName "kube-api-access-5sxrr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:24:07.220212 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:24:07.220181    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d840eb0-a195-4e73-969a-0a478f5b16fd-tls-certs\") pod \"2d840eb0-a195-4e73-969a-0a478f5b16fd\" (UID: \"2d840eb0-a195-4e73-969a-0a478f5b16fd\") "
Aug 04 18:24:07.220377 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:24:07.220360    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5sxrr\" (UniqueName: \"kubernetes.io/projected/2d840eb0-a195-4e73-969a-0a478f5b16fd-kube-api-access-5sxrr\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:24:07.222135 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:24:07.222111    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d840eb0-a195-4e73-969a-0a478f5b16fd-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2d840eb0-a195-4e73-969a-0a478f5b16fd" (UID: "2d840eb0-a195-4e73-969a-0a478f5b16fd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:24:07.320842 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:24:07.320755    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d840eb0-a195-4e73-969a-0a478f5b16fd-tls-certs\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:24:07.891054 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:24:07.891018    2583 generic.go:356] "Generic (PLEG): container finished" podID="2d840eb0-a195-4e73-969a-0a478f5b16fd" containerID="4e7843e01b86192aab3fcb5974bfdc3f20cd7768d6f8d6a0a9e5320a27fbb385" exitCode=0
Aug 04 18:24:07.891515 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:24:07.891075    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedbzrq7"
Aug 04 18:24:07.891515 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:24:07.891096    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedbzrq7" event={"ID":"2d840eb0-a195-4e73-969a-0a478f5b16fd","Type":"ContainerDied","Data":"4e7843e01b86192aab3fcb5974bfdc3f20cd7768d6f8d6a0a9e5320a27fbb385"}
Aug 04 18:24:07.891515 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:24:07.891168    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedbzrq7" event={"ID":"2d840eb0-a195-4e73-969a-0a478f5b16fd","Type":"ContainerDied","Data":"ee18b4e8ff5c7e3c57a608bf1d296f675012b34ff706e8ced058e659ac5b59ea"}
Aug 04 18:24:07.891515 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:24:07.891190    2583 scope.go:117] "RemoveContainer" containerID="4e7843e01b86192aab3fcb5974bfdc3f20cd7768d6f8d6a0a9e5320a27fbb385"
Aug 04 18:24:07.900661 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:24:07.900646    2583 scope.go:117] "RemoveContainer" containerID="4e7843e01b86192aab3fcb5974bfdc3f20cd7768d6f8d6a0a9e5320a27fbb385"
Aug 04 18:24:07.900926 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:24:07.900905    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e7843e01b86192aab3fcb5974bfdc3f20cd7768d6f8d6a0a9e5320a27fbb385\": container with ID starting with 4e7843e01b86192aab3fcb5974bfdc3f20cd7768d6f8d6a0a9e5320a27fbb385 not found: ID does not exist" containerID="4e7843e01b86192aab3fcb5974bfdc3f20cd7768d6f8d6a0a9e5320a27fbb385"
Aug 04 18:24:07.901026 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:24:07.900931    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e7843e01b86192aab3fcb5974bfdc3f20cd7768d6f8d6a0a9e5320a27fbb385"} err="failed to get container status \"4e7843e01b86192aab3fcb5974bfdc3f20cd7768d6f8d6a0a9e5320a27fbb385\": rpc error: code = NotFound desc = could not find container \"4e7843e01b86192aab3fcb5974bfdc3f20cd7768d6f8d6a0a9e5320a27fbb385\": container with ID starting with 4e7843e01b86192aab3fcb5974bfdc3f20cd7768d6f8d6a0a9e5320a27fbb385 not found: ID does not exist"
Aug 04 18:24:07.914237 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:24:07.914209    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedbzrq7"]
Aug 04 18:24:07.917253 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:24:07.917232    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedbzrq7"]
Aug 04 18:24:08.943085 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:24:08.943051    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2d840eb0-a195-4e73-969a-0a478f5b16fd" path="/var/lib/kubelet/pods/2d840eb0-a195-4e73-969a-0a478f5b16fd/volumes"
Aug 04 18:24:18.537662 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:24:18.537626    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-xbsx9"]
Aug 04 18:24:18.538040 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:24:18.537973    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fba83362-7bb6-4c74-b0e5-1e9cf25ac79e" containerName="storage-initializer"
Aug 04 18:24:18.538040 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:24:18.537984    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="fba83362-7bb6-4c74-b0e5-1e9cf25ac79e" containerName="storage-initializer"
Aug 04 18:24:18.538040 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:24:18.538000    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fba83362-7bb6-4c74-b0e5-1e9cf25ac79e" containerName="main"
Aug 04 18:24:18.538040 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:24:18.538005    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="fba83362-7bb6-4c74-b0e5-1e9cf25ac79e" containerName="main"
Aug 04 18:24:18.538040 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:24:18.538013    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d840eb0-a195-4e73-969a-0a478f5b16fd" containerName="main"
Aug 04 18:24:18.538040 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:24:18.538019    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d840eb0-a195-4e73-969a-0a478f5b16fd" containerName="main"
Aug 04 18:24:18.538253 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:24:18.538082    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="fba83362-7bb6-4c74-b0e5-1e9cf25ac79e" containerName="main"
Aug 04 18:24:18.538253 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:24:18.538089    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d840eb0-a195-4e73-969a-0a478f5b16fd" containerName="main"
Aug 04 18:24:18.541121 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:24:18.541102    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-xbsx9"
Aug 04 18:24:18.548049 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:24:18.548030    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-canary-service-backend-7b63470c\"/\"canary-v2-kserve-self-signed-certs\""
Aug 04 18:24:18.549184 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:24:18.549159    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-canary-service-backend-7b63470c\"/\"kube-root-ca.crt\""
Aug 04 18:24:18.553891 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:24:18.553872    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-canary-service-backend-7b63470c\"/\"openshift-service-ca.crt\""
Aug 04 18:24:18.554626 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:24:18.554612    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-canary-service-backend-7b63470c\"/\"default-dockercfg-fdcmj\""
Aug 04 18:24:18.569356 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:24:18.569332    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-xbsx9"]
Aug 04 18:24:18.615161 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:24:18.615129    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0bd3381e-6544-4ad5-86ad-2bdf37e2558c-home\") pod \"canary-v2-kserve-7f65dbbccc-xbsx9\" (UID: \"0bd3381e-6544-4ad5-86ad-2bdf37e2558c\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-xbsx9"
Aug 04 18:24:18.615292 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:24:18.615163    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwvsq\" (UniqueName: \"kubernetes.io/projected/0bd3381e-6544-4ad5-86ad-2bdf37e2558c-kube-api-access-jwvsq\") pod \"canary-v2-kserve-7f65dbbccc-xbsx9\" (UID: \"0bd3381e-6544-4ad5-86ad-2bdf37e2558c\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-xbsx9"
Aug 04 18:24:18.615292 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:24:18.615200    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0bd3381e-6544-4ad5-86ad-2bdf37e2558c-tls-certs\") pod \"canary-v2-kserve-7f65dbbccc-xbsx9\" (UID: \"0bd3381e-6544-4ad5-86ad-2bdf37e2558c\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-xbsx9"
Aug 04 18:24:18.615292 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:24:18.615267    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0bd3381e-6544-4ad5-86ad-2bdf37e2558c-tmp-dir\") pod \"canary-v2-kserve-7f65dbbccc-xbsx9\" (UID: \"0bd3381e-6544-4ad5-86ad-2bdf37e2558c\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-xbsx9"
Aug 04 18:24:18.615450 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:24:18.615301    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0bd3381e-6544-4ad5-86ad-2bdf37e2558c-dshm\") pod \"canary-v2-kserve-7f65dbbccc-xbsx9\" (UID: \"0bd3381e-6544-4ad5-86ad-2bdf37e2558c\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-xbsx9"
Aug 04 18:24:18.615450 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:24:18.615343    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0bd3381e-6544-4ad5-86ad-2bdf37e2558c-model-cache\") pod \"canary-v2-kserve-7f65dbbccc-xbsx9\" (UID: \"0bd3381e-6544-4ad5-86ad-2bdf37e2558c\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-xbsx9"
Aug 04 18:24:18.716143 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:24:18.716113    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0bd3381e-6544-4ad5-86ad-2bdf37e2558c-tmp-dir\") pod \"canary-v2-kserve-7f65dbbccc-xbsx9\" (UID: \"0bd3381e-6544-4ad5-86ad-2bdf37e2558c\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-xbsx9"
Aug 04 18:24:18.716267 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:24:18.716147    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0bd3381e-6544-4ad5-86ad-2bdf37e2558c-dshm\") pod \"canary-v2-kserve-7f65dbbccc-xbsx9\" (UID: \"0bd3381e-6544-4ad5-86ad-2bdf37e2558c\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-xbsx9"
Aug 04 18:24:18.716267 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:24:18.716177    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0bd3381e-6544-4ad5-86ad-2bdf37e2558c-model-cache\") pod \"canary-v2-kserve-7f65dbbccc-xbsx9\" (UID: \"0bd3381e-6544-4ad5-86ad-2bdf37e2558c\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-xbsx9"
Aug 04 18:24:18.716391 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:24:18.716293    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0bd3381e-6544-4ad5-86ad-2bdf37e2558c-home\") pod \"canary-v2-kserve-7f65dbbccc-xbsx9\" (UID: \"0bd3381e-6544-4ad5-86ad-2bdf37e2558c\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-xbsx9"
Aug 04 18:24:18.716391 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:24:18.716325    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwvsq\" (UniqueName: \"kubernetes.io/projected/0bd3381e-6544-4ad5-86ad-2bdf37e2558c-kube-api-access-jwvsq\") pod \"canary-v2-kserve-7f65dbbccc-xbsx9\" (UID: \"0bd3381e-6544-4ad5-86ad-2bdf37e2558c\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-xbsx9"
Aug 04 18:24:18.716391 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:24:18.716363    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0bd3381e-6544-4ad5-86ad-2bdf37e2558c-tls-certs\") pod \"canary-v2-kserve-7f65dbbccc-xbsx9\" (UID: \"0bd3381e-6544-4ad5-86ad-2bdf37e2558c\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-xbsx9"
Aug 04 18:24:18.716615 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:24:18.716593    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0bd3381e-6544-4ad5-86ad-2bdf37e2558c-model-cache\") pod \"canary-v2-kserve-7f65dbbccc-xbsx9\" (UID: \"0bd3381e-6544-4ad5-86ad-2bdf37e2558c\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-xbsx9"
Aug 04 18:24:18.716673 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:24:18.716648    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0bd3381e-6544-4ad5-86ad-2bdf37e2558c-tmp-dir\") pod \"canary-v2-kserve-7f65dbbccc-xbsx9\" (UID: \"0bd3381e-6544-4ad5-86ad-2bdf37e2558c\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-xbsx9"
Aug 04 18:24:18.716747 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:24:18.716669    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0bd3381e-6544-4ad5-86ad-2bdf37e2558c-home\") pod \"canary-v2-kserve-7f65dbbccc-xbsx9\" (UID: \"0bd3381e-6544-4ad5-86ad-2bdf37e2558c\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-xbsx9"
Aug 04 18:24:18.724101 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:24:18.724075    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0bd3381e-6544-4ad5-86ad-2bdf37e2558c-dshm\") pod \"canary-v2-kserve-7f65dbbccc-xbsx9\" (UID: \"0bd3381e-6544-4ad5-86ad-2bdf37e2558c\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-xbsx9"
Aug 04 18:24:18.724811 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:24:18.724789    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0bd3381e-6544-4ad5-86ad-2bdf37e2558c-tls-certs\") pod \"canary-v2-kserve-7f65dbbccc-xbsx9\" (UID: \"0bd3381e-6544-4ad5-86ad-2bdf37e2558c\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-xbsx9"
Aug 04 18:24:18.731485 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:24:18.731447    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwvsq\" (UniqueName: \"kubernetes.io/projected/0bd3381e-6544-4ad5-86ad-2bdf37e2558c-kube-api-access-jwvsq\") pod \"canary-v2-kserve-7f65dbbccc-xbsx9\" (UID: \"0bd3381e-6544-4ad5-86ad-2bdf37e2558c\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-xbsx9"
Aug 04 18:24:18.851157 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:24:18.851089    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-xbsx9"
Aug 04 18:24:19.002932 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:24:19.002903    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-xbsx9"]
Aug 04 18:24:19.009289 ip-10-0-135-124 kubenswrapper[2583]: W0804 18:24:19.009259    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bd3381e_6544_4ad5_86ad_2bdf37e2558c.slice/crio-00ed63a5a7f15659040747843587ec7d6d2f45df5a833ccf6499913bcd654cf6 WatchSource:0}: Error finding container 00ed63a5a7f15659040747843587ec7d6d2f45df5a833ccf6499913bcd654cf6: Status 404 returned error can't find the container with id 00ed63a5a7f15659040747843587ec7d6d2f45df5a833ccf6499913bcd654cf6
Aug 04 18:24:19.011709 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:24:19.011690    2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Aug 04 18:24:19.957317 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:24:19.957283    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-xbsx9" event={"ID":"0bd3381e-6544-4ad5-86ad-2bdf37e2558c","Type":"ContainerStarted","Data":"00ed63a5a7f15659040747843587ec7d6d2f45df5a833ccf6499913bcd654cf6"}
Aug 04 18:24:20.965156 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:24:20.965118    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-xbsx9" event={"ID":"0bd3381e-6544-4ad5-86ad-2bdf37e2558c","Type":"ContainerStarted","Data":"0889f737a66a8498d2eb966872a511b7886302565efd4e302c500f4acc9299cf"}
Aug 04 18:24:21.001096 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:24:21.001041    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-xbsx9" podStartSLOduration=1.995164339 podStartE2EDuration="3.001027193s" totalImagesPullingTime="1.005862854s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:24:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:24:20.999623368 +0000 UTC m=+1474.580841937" watchObservedRunningTime="2026-08-04 18:24:21.001027193 +0000 UTC m=+1474.582245773"
Aug 04 18:24:21.851636 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:24:21.851605    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-xbsx9"
Aug 04 18:24:28.851800 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:24:28.851758    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-xbsx9"
Aug 04 18:24:28.864361 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:24:28.864330    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-xbsx9"
Aug 04 18:24:29.018807 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:24:29.018774    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-xbsx9"
Aug 04 18:24:46.995184 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:24:46.995076    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-wskfb_6157c2d3-2fd0-4544-88ad-434ac75f8c04/console-operator/1.log"
Aug 04 18:24:47.000375 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:24:46.999491    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-wskfb_6157c2d3-2fd0-4544-88ad-434ac75f8c04/console-operator/1.log"
Aug 04 18:25:14.846392 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:25:14.846311    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-56649fd646-mwqdj"]
Aug 04 18:25:14.850079 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:25:14.850059    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-56649fd646-mwqdj"
Aug 04 18:25:14.853768 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:25:14.853746    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"seaweedfs-s3-creds\""
Aug 04 18:25:14.853875 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:25:14.853773    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"tokenizer-clean-path-test-kserve-self-signed-certs\""
Aug 04 18:25:14.855188 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:25:14.855169    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"default-dockercfg-cjq6s\""
Aug 04 18:25:14.855188 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:25:14.855180    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"openshift-service-ca.crt\""
Aug 04 18:25:14.855342 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:25:14.855184    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"odh-kserve-custom-ca-bundle\""
Aug 04 18:25:14.855342 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:25:14.855224    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"kube-root-ca.crt\""
Aug 04 18:25:14.866061 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:25:14.866039    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-56649fd646-mwqdj"]
Aug 04 18:25:14.884131 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:25:14.884105    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/83d92453-9393-44cc-885a-403f1c5a6ba5-model-cache\") pod \"tokenizer-clean-path-test-kserve-56649fd646-mwqdj\" (UID: \"83d92453-9393-44cc-885a-403f1c5a6ba5\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-56649fd646-mwqdj"
Aug 04 18:25:14.884230 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:25:14.884135    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/83d92453-9393-44cc-885a-403f1c5a6ba5-dshm\") pod \"tokenizer-clean-path-test-kserve-56649fd646-mwqdj\" (UID: \"83d92453-9393-44cc-885a-403f1c5a6ba5\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-56649fd646-mwqdj"
Aug 04 18:25:14.884230 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:25:14.884205    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/83d92453-9393-44cc-885a-403f1c5a6ba5-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-56649fd646-mwqdj\" (UID: \"83d92453-9393-44cc-885a-403f1c5a6ba5\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-56649fd646-mwqdj"
Aug 04 18:25:14.884347 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:25:14.884234    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbk98\" (UniqueName: \"kubernetes.io/projected/83d92453-9393-44cc-885a-403f1c5a6ba5-kube-api-access-bbk98\") pod \"tokenizer-clean-path-test-kserve-56649fd646-mwqdj\" (UID: \"83d92453-9393-44cc-885a-403f1c5a6ba5\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-56649fd646-mwqdj"
Aug 04 18:25:14.884347 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:25:14.884301    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/83d92453-9393-44cc-885a-403f1c5a6ba5-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-56649fd646-mwqdj\" (UID: \"83d92453-9393-44cc-885a-403f1c5a6ba5\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-56649fd646-mwqdj"
Aug 04 18:25:14.884347 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:25:14.884341    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/83d92453-9393-44cc-885a-403f1c5a6ba5-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-56649fd646-mwqdj\" (UID: \"83d92453-9393-44cc-885a-403f1c5a6ba5\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-56649fd646-mwqdj"
Aug 04 18:25:14.884454 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:25:14.884365    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/83d92453-9393-44cc-885a-403f1c5a6ba5-home\") pod \"tokenizer-clean-path-test-kserve-56649fd646-mwqdj\" (UID: \"83d92453-9393-44cc-885a-403f1c5a6ba5\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-56649fd646-mwqdj"
Aug 04 18:25:14.884454 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:25:14.884384    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/83d92453-9393-44cc-885a-403f1c5a6ba5-tls-certs\") pod \"tokenizer-clean-path-test-kserve-56649fd646-mwqdj\" (UID: \"83d92453-9393-44cc-885a-403f1c5a6ba5\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-56649fd646-mwqdj"
Aug 04 18:25:14.985418 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:25:14.985387    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/83d92453-9393-44cc-885a-403f1c5a6ba5-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-56649fd646-mwqdj\" (UID: \"83d92453-9393-44cc-885a-403f1c5a6ba5\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-56649fd646-mwqdj"
Aug 04 18:25:14.985615 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:25:14.985423    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbk98\" (UniqueName: \"kubernetes.io/projected/83d92453-9393-44cc-885a-403f1c5a6ba5-kube-api-access-bbk98\") pod \"tokenizer-clean-path-test-kserve-56649fd646-mwqdj\" (UID: \"83d92453-9393-44cc-885a-403f1c5a6ba5\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-56649fd646-mwqdj"
Aug 04 18:25:14.985615 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:25:14.985452    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/83d92453-9393-44cc-885a-403f1c5a6ba5-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-56649fd646-mwqdj\" (UID: \"83d92453-9393-44cc-885a-403f1c5a6ba5\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-56649fd646-mwqdj"
Aug 04 18:25:14.985615 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:25:14.985490    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/83d92453-9393-44cc-885a-403f1c5a6ba5-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-56649fd646-mwqdj\" (UID: \"83d92453-9393-44cc-885a-403f1c5a6ba5\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-56649fd646-mwqdj"
Aug 04 18:25:14.985615 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:25:14.985510    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/83d92453-9393-44cc-885a-403f1c5a6ba5-home\") pod \"tokenizer-clean-path-test-kserve-56649fd646-mwqdj\" (UID: \"83d92453-9393-44cc-885a-403f1c5a6ba5\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-56649fd646-mwqdj"
Aug 04 18:25:14.985832 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:25:14.985626    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/83d92453-9393-44cc-885a-403f1c5a6ba5-tls-certs\") pod \"tokenizer-clean-path-test-kserve-56649fd646-mwqdj\" (UID: \"83d92453-9393-44cc-885a-403f1c5a6ba5\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-56649fd646-mwqdj"
Aug 04 18:25:14.985832 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:25:14.985680    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/83d92453-9393-44cc-885a-403f1c5a6ba5-model-cache\") pod \"tokenizer-clean-path-test-kserve-56649fd646-mwqdj\" (UID: \"83d92453-9393-44cc-885a-403f1c5a6ba5\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-56649fd646-mwqdj"
Aug 04 18:25:14.985832 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:25:14.985707    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/83d92453-9393-44cc-885a-403f1c5a6ba5-dshm\") pod \"tokenizer-clean-path-test-kserve-56649fd646-mwqdj\" (UID: \"83d92453-9393-44cc-885a-403f1c5a6ba5\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-56649fd646-mwqdj"
Aug 04 18:25:14.985986 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:25:14.985877    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/83d92453-9393-44cc-885a-403f1c5a6ba5-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-56649fd646-mwqdj\" (UID: \"83d92453-9393-44cc-885a-403f1c5a6ba5\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-56649fd646-mwqdj"
Aug 04 18:25:14.985986 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:25:14.985965    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/83d92453-9393-44cc-885a-403f1c5a6ba5-home\") pod \"tokenizer-clean-path-test-kserve-56649fd646-mwqdj\" (UID: \"83d92453-9393-44cc-885a-403f1c5a6ba5\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-56649fd646-mwqdj"
Aug 04 18:25:14.986084 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:25:14.986009    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/83d92453-9393-44cc-885a-403f1c5a6ba5-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-56649fd646-mwqdj\" (UID: \"83d92453-9393-44cc-885a-403f1c5a6ba5\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-56649fd646-mwqdj"
Aug 04 18:25:14.986084 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:25:14.986074    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/83d92453-9393-44cc-885a-403f1c5a6ba5-model-cache\") pod \"tokenizer-clean-path-test-kserve-56649fd646-mwqdj\" (UID: \"83d92453-9393-44cc-885a-403f1c5a6ba5\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-56649fd646-mwqdj"
Aug 04 18:25:14.986202 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:25:14.986181    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/83d92453-9393-44cc-885a-403f1c5a6ba5-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-56649fd646-mwqdj\" (UID: \"83d92453-9393-44cc-885a-403f1c5a6ba5\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-56649fd646-mwqdj"
Aug 04 18:25:14.987869 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:25:14.987844    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/83d92453-9393-44cc-885a-403f1c5a6ba5-dshm\") pod \"tokenizer-clean-path-test-kserve-56649fd646-mwqdj\" (UID: \"83d92453-9393-44cc-885a-403f1c5a6ba5\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-56649fd646-mwqdj"
Aug 04 18:25:14.988152 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:25:14.988134    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/83d92453-9393-44cc-885a-403f1c5a6ba5-tls-certs\") pod \"tokenizer-clean-path-test-kserve-56649fd646-mwqdj\" (UID: \"83d92453-9393-44cc-885a-403f1c5a6ba5\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-56649fd646-mwqdj"
Aug 04 18:25:15.001923 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:25:15.001897    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbk98\" (UniqueName: \"kubernetes.io/projected/83d92453-9393-44cc-885a-403f1c5a6ba5-kube-api-access-bbk98\") pod \"tokenizer-clean-path-test-kserve-56649fd646-mwqdj\" (UID: \"83d92453-9393-44cc-885a-403f1c5a6ba5\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-56649fd646-mwqdj"
Aug 04 18:25:15.144989 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:25:15.144916    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-68d5485bbc-lkdmg"]
Aug 04 18:25:15.149580 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:25:15.149560    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-68d5485bbc-lkdmg"
Aug 04 18:25:15.160736 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:25:15.160715    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-56649fd646-mwqdj"
Aug 04 18:25:15.183606 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:25:15.183573    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-68d5485bbc-lkdmg"]
Aug 04 18:25:15.189872 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:25:15.189838    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18fc9d06-f3fa-4437-b889-a9e2b2fb767d-tls-certs\") pod \"tokenizer-clean-path-test-tokenizer-68d5485bbc-lkdmg\" (UID: \"18fc9d06-f3fa-4437-b889-a9e2b2fb767d\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-68d5485bbc-lkdmg"
Aug 04 18:25:15.189998 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:25:15.189906    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18fc9d06-f3fa-4437-b889-a9e2b2fb767d-tmp-dir\") pod \"tokenizer-clean-path-test-tokenizer-68d5485bbc-lkdmg\" (UID: \"18fc9d06-f3fa-4437-b889-a9e2b2fb767d\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-68d5485bbc-lkdmg"
Aug 04 18:25:15.189998 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:25:15.189958    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/18fc9d06-f3fa-4437-b889-a9e2b2fb767d-kserve-provision-location\") pod \"tokenizer-clean-path-test-tokenizer-68d5485bbc-lkdmg\" (UID: \"18fc9d06-f3fa-4437-b889-a9e2b2fb767d\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-68d5485bbc-lkdmg"
Aug 04 18:25:15.190119 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:25:15.190000    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/18fc9d06-f3fa-4437-b889-a9e2b2fb767d-cabundle-cert\") pod \"tokenizer-clean-path-test-tokenizer-68d5485bbc-lkdmg\" (UID: \"18fc9d06-f3fa-4437-b889-a9e2b2fb767d\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-68d5485bbc-lkdmg"
Aug 04 18:25:15.291202 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:25:15.291162    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18fc9d06-f3fa-4437-b889-a9e2b2fb767d-tls-certs\") pod \"tokenizer-clean-path-test-tokenizer-68d5485bbc-lkdmg\" (UID: \"18fc9d06-f3fa-4437-b889-a9e2b2fb767d\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-68d5485bbc-lkdmg"
Aug 04 18:25:15.291551 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:25:15.291220    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18fc9d06-f3fa-4437-b889-a9e2b2fb767d-tmp-dir\") pod \"tokenizer-clean-path-test-tokenizer-68d5485bbc-lkdmg\" (UID: \"18fc9d06-f3fa-4437-b889-a9e2b2fb767d\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-68d5485bbc-lkdmg"
Aug 04 18:25:15.291551 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:25:15.291263    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/18fc9d06-f3fa-4437-b889-a9e2b2fb767d-kserve-provision-location\") pod \"tokenizer-clean-path-test-tokenizer-68d5485bbc-lkdmg\" (UID: \"18fc9d06-f3fa-4437-b889-a9e2b2fb767d\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-68d5485bbc-lkdmg"
Aug 04 18:25:15.291551 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:25:15.291299    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/18fc9d06-f3fa-4437-b889-a9e2b2fb767d-cabundle-cert\") pod \"tokenizer-clean-path-test-tokenizer-68d5485bbc-lkdmg\" (UID: \"18fc9d06-f3fa-4437-b889-a9e2b2fb767d\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-68d5485bbc-lkdmg"
Aug 04 18:25:15.292033 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:25:15.291741    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/18fc9d06-f3fa-4437-b889-a9e2b2fb767d-kserve-provision-location\") pod \"tokenizer-clean-path-test-tokenizer-68d5485bbc-lkdmg\" (UID: \"18fc9d06-f3fa-4437-b889-a9e2b2fb767d\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-68d5485bbc-lkdmg"
Aug 04 18:25:15.292033 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:25:15.291843    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18fc9d06-f3fa-4437-b889-a9e2b2fb767d-tmp-dir\") pod \"tokenizer-clean-path-test-tokenizer-68d5485bbc-lkdmg\" (UID: \"18fc9d06-f3fa-4437-b889-a9e2b2fb767d\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-68d5485bbc-lkdmg"
Aug 04 18:25:15.292269 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:25:15.292238    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/18fc9d06-f3fa-4437-b889-a9e2b2fb767d-cabundle-cert\") pod \"tokenizer-clean-path-test-tokenizer-68d5485bbc-lkdmg\" (UID: \"18fc9d06-f3fa-4437-b889-a9e2b2fb767d\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-68d5485bbc-lkdmg"
Aug 04 18:25:15.294064 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:25:15.294037    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18fc9d06-f3fa-4437-b889-a9e2b2fb767d-tls-certs\") pod \"tokenizer-clean-path-test-tokenizer-68d5485bbc-lkdmg\" (UID: \"18fc9d06-f3fa-4437-b889-a9e2b2fb767d\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-68d5485bbc-lkdmg"
Aug 04 18:25:15.324114 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:25:15.324087    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-56649fd646-mwqdj"]
Aug 04 18:25:15.325363 ip-10-0-135-124 kubenswrapper[2583]: W0804 18:25:15.325340    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83d92453_9393_44cc_885a_403f1c5a6ba5.slice/crio-056749a6fec82af2fc570a03a91591246ebdbaea27b6d9c3a22bbb026c8e9718 WatchSource:0}: Error finding container 056749a6fec82af2fc570a03a91591246ebdbaea27b6d9c3a22bbb026c8e9718: Status 404 returned error can't find the container with id 056749a6fec82af2fc570a03a91591246ebdbaea27b6d9c3a22bbb026c8e9718
Aug 04 18:25:15.460626 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:25:15.460595    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-68d5485bbc-lkdmg"
Aug 04 18:25:15.615039 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:25:15.615017    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-68d5485bbc-lkdmg"]
Aug 04 18:25:15.617153 ip-10-0-135-124 kubenswrapper[2583]: W0804 18:25:15.617126    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18fc9d06_f3fa_4437_b889_a9e2b2fb767d.slice/crio-50d1a456507791279740d631cf706142243d9ebbf0f0ebab050f1fe54bbd7649 WatchSource:0}: Error finding container 50d1a456507791279740d631cf706142243d9ebbf0f0ebab050f1fe54bbd7649: Status 404 returned error can't find the container with id 50d1a456507791279740d631cf706142243d9ebbf0f0ebab050f1fe54bbd7649
Aug 04 18:25:16.262842 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:25:16.262800    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-56649fd646-mwqdj" event={"ID":"83d92453-9393-44cc-885a-403f1c5a6ba5","Type":"ContainerStarted","Data":"c2b9da762d15a68359431e05010b9d8738425b4c2f644d5c77d46fd9e56dc03f"}
Aug 04 18:25:16.263415 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:25:16.262862    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-56649fd646-mwqdj" event={"ID":"83d92453-9393-44cc-885a-403f1c5a6ba5","Type":"ContainerStarted","Data":"056749a6fec82af2fc570a03a91591246ebdbaea27b6d9c3a22bbb026c8e9718"}
Aug 04 18:25:16.264414 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:25:16.264383    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-68d5485bbc-lkdmg" event={"ID":"18fc9d06-f3fa-4437-b889-a9e2b2fb767d","Type":"ContainerStarted","Data":"fd6a4fbbe13fc1529a4de83891b7b5cce1c3e3b21e9edb95475c8867d694481f"}
Aug 04 18:25:16.264571 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:25:16.264446    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-68d5485bbc-lkdmg" event={"ID":"18fc9d06-f3fa-4437-b889-a9e2b2fb767d","Type":"ContainerStarted","Data":"50d1a456507791279740d631cf706142243d9ebbf0f0ebab050f1fe54bbd7649"}
Aug 04 18:25:18.278847 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:25:18.278814    2583 generic.go:356] "Generic (PLEG): container finished" podID="83d92453-9393-44cc-885a-403f1c5a6ba5" containerID="c2b9da762d15a68359431e05010b9d8738425b4c2f644d5c77d46fd9e56dc03f" exitCode=0
Aug 04 18:25:18.279262 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:25:18.278883    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-56649fd646-mwqdj" event={"ID":"83d92453-9393-44cc-885a-403f1c5a6ba5","Type":"ContainerDied","Data":"c2b9da762d15a68359431e05010b9d8738425b4c2f644d5c77d46fd9e56dc03f"}
Aug 04 18:25:18.280429 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:25:18.280407    2583 generic.go:356] "Generic (PLEG): container finished" podID="18fc9d06-f3fa-4437-b889-a9e2b2fb767d" containerID="fd6a4fbbe13fc1529a4de83891b7b5cce1c3e3b21e9edb95475c8867d694481f" exitCode=0
Aug 04 18:25:18.280560 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:25:18.280461    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-68d5485bbc-lkdmg" event={"ID":"18fc9d06-f3fa-4437-b889-a9e2b2fb767d","Type":"ContainerDied","Data":"fd6a4fbbe13fc1529a4de83891b7b5cce1c3e3b21e9edb95475c8867d694481f"}
Aug 04 18:25:19.289945 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:25:19.289906    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-56649fd646-mwqdj" event={"ID":"83d92453-9393-44cc-885a-403f1c5a6ba5","Type":"ContainerStarted","Data":"07091bdaa70a10e534c4122b5350b1ce0b76b3e457cc7c839285f9f1fb5ed20f"}
Aug 04 18:25:19.317251 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:25:19.317183    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-56649fd646-mwqdj" podStartSLOduration=2.855415696 podStartE2EDuration="5.317159739s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.461744043s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:25:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:25:19.312412017 +0000 UTC m=+1532.893630596" watchObservedRunningTime="2026-08-04 18:25:19.317159739 +0000 UTC m=+1532.898378318"
Aug 04 18:25:20.161840 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:25:20.161796    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-56649fd646-mwqdj"
Aug 04 18:25:25.161632 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:25:25.161598    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-56649fd646-mwqdj"
Aug 04 18:25:25.173993 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:25:25.173970    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-56649fd646-mwqdj"
Aug 04 18:25:25.336492 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:25:25.336443    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-56649fd646-mwqdj"
Aug 04 18:25:58.535523 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:25:58.535491    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-68d5485bbc-lkdmg" event={"ID":"18fc9d06-f3fa-4437-b889-a9e2b2fb767d","Type":"ContainerStarted","Data":"e76febed948f35df2fda7564f886a26f4232d64aee5f448e4e6312f9efc8c67c"}
Aug 04 18:25:58.535919 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:25:58.535691    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-68d5485bbc-lkdmg"
Aug 04 18:25:58.560220 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:25:58.560179    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-68d5485bbc-lkdmg" podStartSLOduration=0.95392657 podStartE2EDuration="43.560167474s" totalImagesPullingTime="40.15055949s" totalInitContainerRuntime="2.455681414s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:25:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:25:58.557310618 +0000 UTC m=+1572.138529196" watchObservedRunningTime="2026-08-04 18:25:58.560167474 +0000 UTC m=+1572.141386052"
Aug 04 18:26:29.552831 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:26:29.552794    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-68d5485bbc-lkdmg"
Aug 04 18:26:30.443532 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:26:30.443497    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-cf47448b_tokenizer-clean-path-test-tokenizer-68d5485bbc-lkdmg_18fc9d06-f3fa-4437-b889-a9e2b2fb767d/main/0.log"
Aug 04 18:26:30.957243 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:26:30.957210    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-68d5485bbc-lkdmg"]
Aug 04 18:26:30.957736 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:26:30.957395    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-68d5485bbc-lkdmg" podUID="18fc9d06-f3fa-4437-b889-a9e2b2fb767d" containerName="main" containerID="cri-o://e76febed948f35df2fda7564f886a26f4232d64aee5f448e4e6312f9efc8c67c" gracePeriod=30
Aug 04 18:26:30.971641 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:26:30.971606    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-56649fd646-mwqdj"]
Aug 04 18:26:30.981121 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:26:30.981083    2583 log.go:32] "ExecSync cmd from runtime service failed" err=<
Aug 04 18:26:30.981121 ip-10-0-135-124 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Aug 04 18:26:30.981121 ip-10-0-135-124 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Aug 04 18:26:30.981121 ip-10-0-135-124 kubenswrapper[2583]:  > containerID="07091bdaa70a10e534c4122b5350b1ce0b76b3e457cc7c839285f9f1fb5ed20f" cmd=["/bin/sleep","15"]
Aug 04 18:26:30.981432 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:26:30.981137    2583 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-56649fd646-mwqdj" message="" err=<
Aug 04 18:26:30.981432 ip-10-0-135-124 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Aug 04 18:26:30.981432 ip-10-0-135-124 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Aug 04 18:26:30.981432 ip-10-0-135-124 kubenswrapper[2583]:  >
Aug 04 18:26:30.981432 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:26:30.981152    2583 kuberuntime_container.go:755] "PreStop hook failed" err=<
Aug 04 18:26:30.981432 ip-10-0-135-124 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Aug 04 18:26:30.981432 ip-10-0-135-124 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Aug 04 18:26:30.981432 ip-10-0-135-124 kubenswrapper[2583]:  > pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-56649fd646-mwqdj" podUID="83d92453-9393-44cc-885a-403f1c5a6ba5" containerName="main" containerID="cri-o://07091bdaa70a10e534c4122b5350b1ce0b76b3e457cc7c839285f9f1fb5ed20f"
Aug 04 18:26:30.981432 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:26:30.981187    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-56649fd646-mwqdj" podUID="83d92453-9393-44cc-885a-403f1c5a6ba5" containerName="main" containerID="cri-o://07091bdaa70a10e534c4122b5350b1ce0b76b3e457cc7c839285f9f1fb5ed20f" gracePeriod=60
Aug 04 18:26:31.349886 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:26:31.349860    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-56649fd646-mwqdj"
Aug 04 18:26:31.495018 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:26:31.494988    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/83d92453-9393-44cc-885a-403f1c5a6ba5-dshm\") pod \"83d92453-9393-44cc-885a-403f1c5a6ba5\" (UID: \"83d92453-9393-44cc-885a-403f1c5a6ba5\") "
Aug 04 18:26:31.495155 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:26:31.495037    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/83d92453-9393-44cc-885a-403f1c5a6ba5-tmp-dir\") pod \"83d92453-9393-44cc-885a-403f1c5a6ba5\" (UID: \"83d92453-9393-44cc-885a-403f1c5a6ba5\") "
Aug 04 18:26:31.495155 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:26:31.495085    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/83d92453-9393-44cc-885a-403f1c5a6ba5-model-cache\") pod \"83d92453-9393-44cc-885a-403f1c5a6ba5\" (UID: \"83d92453-9393-44cc-885a-403f1c5a6ba5\") "
Aug 04 18:26:31.495155 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:26:31.495108    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/83d92453-9393-44cc-885a-403f1c5a6ba5-cabundle-cert\") pod \"83d92453-9393-44cc-885a-403f1c5a6ba5\" (UID: \"83d92453-9393-44cc-885a-403f1c5a6ba5\") "
Aug 04 18:26:31.495155 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:26:31.495141    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/83d92453-9393-44cc-885a-403f1c5a6ba5-tls-certs\") pod \"83d92453-9393-44cc-885a-403f1c5a6ba5\" (UID: \"83d92453-9393-44cc-885a-403f1c5a6ba5\") "
Aug 04 18:26:31.495371 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:26:31.495168    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/83d92453-9393-44cc-885a-403f1c5a6ba5-home\") pod \"83d92453-9393-44cc-885a-403f1c5a6ba5\" (UID: \"83d92453-9393-44cc-885a-403f1c5a6ba5\") "
Aug 04 18:26:31.495371 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:26:31.495222    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/83d92453-9393-44cc-885a-403f1c5a6ba5-kserve-provision-location\") pod \"83d92453-9393-44cc-885a-403f1c5a6ba5\" (UID: \"83d92453-9393-44cc-885a-403f1c5a6ba5\") "
Aug 04 18:26:31.495371 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:26:31.495259    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bbk98\" (UniqueName: \"kubernetes.io/projected/83d92453-9393-44cc-885a-403f1c5a6ba5-kube-api-access-bbk98\") pod \"83d92453-9393-44cc-885a-403f1c5a6ba5\" (UID: \"83d92453-9393-44cc-885a-403f1c5a6ba5\") "
Aug 04 18:26:31.495548 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:26:31.495414    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83d92453-9393-44cc-885a-403f1c5a6ba5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "83d92453-9393-44cc-885a-403f1c5a6ba5" (UID: "83d92453-9393-44cc-885a-403f1c5a6ba5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:26:31.495548 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:26:31.495459    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83d92453-9393-44cc-885a-403f1c5a6ba5-home" (OuterVolumeSpecName: "home") pod "83d92453-9393-44cc-885a-403f1c5a6ba5" (UID: "83d92453-9393-44cc-885a-403f1c5a6ba5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:26:31.495548 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:26:31.495522    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83d92453-9393-44cc-885a-403f1c5a6ba5-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "83d92453-9393-44cc-885a-403f1c5a6ba5" (UID: "83d92453-9393-44cc-885a-403f1c5a6ba5"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:26:31.495548 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:26:31.495422    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83d92453-9393-44cc-885a-403f1c5a6ba5-model-cache" (OuterVolumeSpecName: "model-cache") pod "83d92453-9393-44cc-885a-403f1c5a6ba5" (UID: "83d92453-9393-44cc-885a-403f1c5a6ba5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:26:31.495766 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:26:31.495734    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/83d92453-9393-44cc-885a-403f1c5a6ba5-tmp-dir\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:26:31.495766 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:26:31.495756    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/83d92453-9393-44cc-885a-403f1c5a6ba5-model-cache\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:26:31.495842 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:26:31.495771    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/83d92453-9393-44cc-885a-403f1c5a6ba5-cabundle-cert\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:26:31.495842 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:26:31.495786    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/83d92453-9393-44cc-885a-403f1c5a6ba5-home\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:26:31.497178 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:26:31.497152    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83d92453-9393-44cc-885a-403f1c5a6ba5-dshm" (OuterVolumeSpecName: "dshm") pod "83d92453-9393-44cc-885a-403f1c5a6ba5" (UID: "83d92453-9393-44cc-885a-403f1c5a6ba5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:26:31.497785 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:26:31.497747    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83d92453-9393-44cc-885a-403f1c5a6ba5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "83d92453-9393-44cc-885a-403f1c5a6ba5" (UID: "83d92453-9393-44cc-885a-403f1c5a6ba5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:26:31.497919 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:26:31.497894    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83d92453-9393-44cc-885a-403f1c5a6ba5-kube-api-access-bbk98" (OuterVolumeSpecName: "kube-api-access-bbk98") pod "83d92453-9393-44cc-885a-403f1c5a6ba5" (UID: "83d92453-9393-44cc-885a-403f1c5a6ba5"). InnerVolumeSpecName "kube-api-access-bbk98". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:26:31.522153 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:26:31.522127    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83d92453-9393-44cc-885a-403f1c5a6ba5-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "83d92453-9393-44cc-885a-403f1c5a6ba5" (UID: "83d92453-9393-44cc-885a-403f1c5a6ba5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:26:31.596281 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:26:31.596250    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/83d92453-9393-44cc-885a-403f1c5a6ba5-kserve-provision-location\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:26:31.596398 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:26:31.596284    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bbk98\" (UniqueName: \"kubernetes.io/projected/83d92453-9393-44cc-885a-403f1c5a6ba5-kube-api-access-bbk98\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:26:31.596398 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:26:31.596299    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/83d92453-9393-44cc-885a-403f1c5a6ba5-dshm\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:26:31.596398 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:26:31.596314    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/83d92453-9393-44cc-885a-403f1c5a6ba5-tls-certs\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:26:31.599172 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:26:31.599152    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-68d5485bbc-lkdmg"
Aug 04 18:26:31.696913 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:26:31.696882    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18fc9d06-f3fa-4437-b889-a9e2b2fb767d-tmp-dir\") pod \"18fc9d06-f3fa-4437-b889-a9e2b2fb767d\" (UID: \"18fc9d06-f3fa-4437-b889-a9e2b2fb767d\") "
Aug 04 18:26:31.697036 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:26:31.696939    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/18fc9d06-f3fa-4437-b889-a9e2b2fb767d-kserve-provision-location\") pod \"18fc9d06-f3fa-4437-b889-a9e2b2fb767d\" (UID: \"18fc9d06-f3fa-4437-b889-a9e2b2fb767d\") "
Aug 04 18:26:31.697036 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:26:31.697001    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/18fc9d06-f3fa-4437-b889-a9e2b2fb767d-cabundle-cert\") pod \"18fc9d06-f3fa-4437-b889-a9e2b2fb767d\" (UID: \"18fc9d06-f3fa-4437-b889-a9e2b2fb767d\") "
Aug 04 18:26:31.697036 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:26:31.697019    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18fc9d06-f3fa-4437-b889-a9e2b2fb767d-tls-certs\") pod \"18fc9d06-f3fa-4437-b889-a9e2b2fb767d\" (UID: \"18fc9d06-f3fa-4437-b889-a9e2b2fb767d\") "
Aug 04 18:26:31.697251 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:26:31.697225    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18fc9d06-f3fa-4437-b889-a9e2b2fb767d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "18fc9d06-f3fa-4437-b889-a9e2b2fb767d" (UID: "18fc9d06-f3fa-4437-b889-a9e2b2fb767d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:26:31.697358 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:26:31.697336    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18fc9d06-f3fa-4437-b889-a9e2b2fb767d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "18fc9d06-f3fa-4437-b889-a9e2b2fb767d" (UID: "18fc9d06-f3fa-4437-b889-a9e2b2fb767d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:26:31.699207 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:26:31.699176    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18fc9d06-f3fa-4437-b889-a9e2b2fb767d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "18fc9d06-f3fa-4437-b889-a9e2b2fb767d" (UID: "18fc9d06-f3fa-4437-b889-a9e2b2fb767d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:26:31.720833 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:26:31.720806    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18fc9d06-f3fa-4437-b889-a9e2b2fb767d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "18fc9d06-f3fa-4437-b889-a9e2b2fb767d" (UID: "18fc9d06-f3fa-4437-b889-a9e2b2fb767d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:26:31.733698 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:26:31.733670    2583 generic.go:356] "Generic (PLEG): container finished" podID="83d92453-9393-44cc-885a-403f1c5a6ba5" containerID="07091bdaa70a10e534c4122b5350b1ce0b76b3e457cc7c839285f9f1fb5ed20f" exitCode=0
Aug 04 18:26:31.733810 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:26:31.733729    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-56649fd646-mwqdj" event={"ID":"83d92453-9393-44cc-885a-403f1c5a6ba5","Type":"ContainerDied","Data":"07091bdaa70a10e534c4122b5350b1ce0b76b3e457cc7c839285f9f1fb5ed20f"}
Aug 04 18:26:31.733810 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:26:31.733805    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-56649fd646-mwqdj" event={"ID":"83d92453-9393-44cc-885a-403f1c5a6ba5","Type":"ContainerDied","Data":"056749a6fec82af2fc570a03a91591246ebdbaea27b6d9c3a22bbb026c8e9718"}
Aug 04 18:26:31.733908 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:26:31.733737    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-56649fd646-mwqdj"
Aug 04 18:26:31.733908 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:26:31.733830    2583 scope.go:117] "RemoveContainer" containerID="07091bdaa70a10e534c4122b5350b1ce0b76b3e457cc7c839285f9f1fb5ed20f"
Aug 04 18:26:31.736004 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:26:31.735979    2583 generic.go:356] "Generic (PLEG): container finished" podID="18fc9d06-f3fa-4437-b889-a9e2b2fb767d" containerID="e76febed948f35df2fda7564f886a26f4232d64aee5f448e4e6312f9efc8c67c" exitCode=0
Aug 04 18:26:31.736120 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:26:31.736027    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-68d5485bbc-lkdmg" event={"ID":"18fc9d06-f3fa-4437-b889-a9e2b2fb767d","Type":"ContainerDied","Data":"e76febed948f35df2fda7564f886a26f4232d64aee5f448e4e6312f9efc8c67c"}
Aug 04 18:26:31.736120 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:26:31.736036    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-68d5485bbc-lkdmg"
Aug 04 18:26:31.736120 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:26:31.736070    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-68d5485bbc-lkdmg" event={"ID":"18fc9d06-f3fa-4437-b889-a9e2b2fb767d","Type":"ContainerDied","Data":"50d1a456507791279740d631cf706142243d9ebbf0f0ebab050f1fe54bbd7649"}
Aug 04 18:26:31.745648 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:26:31.745624    2583 scope.go:117] "RemoveContainer" containerID="c2b9da762d15a68359431e05010b9d8738425b4c2f644d5c77d46fd9e56dc03f"
Aug 04 18:26:31.754044 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:26:31.754029    2583 scope.go:117] "RemoveContainer" containerID="07091bdaa70a10e534c4122b5350b1ce0b76b3e457cc7c839285f9f1fb5ed20f"
Aug 04 18:26:31.754281 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:26:31.754265    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07091bdaa70a10e534c4122b5350b1ce0b76b3e457cc7c839285f9f1fb5ed20f\": container with ID starting with 07091bdaa70a10e534c4122b5350b1ce0b76b3e457cc7c839285f9f1fb5ed20f not found: ID does not exist" containerID="07091bdaa70a10e534c4122b5350b1ce0b76b3e457cc7c839285f9f1fb5ed20f"
Aug 04 18:26:31.754326 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:26:31.754289    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07091bdaa70a10e534c4122b5350b1ce0b76b3e457cc7c839285f9f1fb5ed20f"} err="failed to get container status \"07091bdaa70a10e534c4122b5350b1ce0b76b3e457cc7c839285f9f1fb5ed20f\": rpc error: code = NotFound desc = could not find container \"07091bdaa70a10e534c4122b5350b1ce0b76b3e457cc7c839285f9f1fb5ed20f\": container with ID starting with 07091bdaa70a10e534c4122b5350b1ce0b76b3e457cc7c839285f9f1fb5ed20f not found: ID does not exist"
Aug 04 18:26:31.754326 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:26:31.754304    2583 scope.go:117] "RemoveContainer" containerID="c2b9da762d15a68359431e05010b9d8738425b4c2f644d5c77d46fd9e56dc03f"
Aug 04 18:26:31.754612 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:26:31.754591    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2b9da762d15a68359431e05010b9d8738425b4c2f644d5c77d46fd9e56dc03f\": container with ID starting with c2b9da762d15a68359431e05010b9d8738425b4c2f644d5c77d46fd9e56dc03f not found: ID does not exist" containerID="c2b9da762d15a68359431e05010b9d8738425b4c2f644d5c77d46fd9e56dc03f"
Aug 04 18:26:31.754655 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:26:31.754619    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2b9da762d15a68359431e05010b9d8738425b4c2f644d5c77d46fd9e56dc03f"} err="failed to get container status \"c2b9da762d15a68359431e05010b9d8738425b4c2f644d5c77d46fd9e56dc03f\": rpc error: code = NotFound desc = could not find container \"c2b9da762d15a68359431e05010b9d8738425b4c2f644d5c77d46fd9e56dc03f\": container with ID starting with c2b9da762d15a68359431e05010b9d8738425b4c2f644d5c77d46fd9e56dc03f not found: ID does not exist"
Aug 04 18:26:31.754655 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:26:31.754636    2583 scope.go:117] "RemoveContainer" containerID="e76febed948f35df2fda7564f886a26f4232d64aee5f448e4e6312f9efc8c67c"
Aug 04 18:26:31.762463 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:26:31.762448    2583 scope.go:117] "RemoveContainer" containerID="fd6a4fbbe13fc1529a4de83891b7b5cce1c3e3b21e9edb95475c8867d694481f"
Aug 04 18:26:31.763721 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:26:31.763703    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-68d5485bbc-lkdmg"]
Aug 04 18:26:31.769386 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:26:31.769366    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-68d5485bbc-lkdmg"]
Aug 04 18:26:31.770883 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:26:31.770866    2583 scope.go:117] "RemoveContainer" containerID="e76febed948f35df2fda7564f886a26f4232d64aee5f448e4e6312f9efc8c67c"
Aug 04 18:26:31.771145 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:26:31.771124    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e76febed948f35df2fda7564f886a26f4232d64aee5f448e4e6312f9efc8c67c\": container with ID starting with e76febed948f35df2fda7564f886a26f4232d64aee5f448e4e6312f9efc8c67c not found: ID does not exist" containerID="e76febed948f35df2fda7564f886a26f4232d64aee5f448e4e6312f9efc8c67c"
Aug 04 18:26:31.771209 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:26:31.771152    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e76febed948f35df2fda7564f886a26f4232d64aee5f448e4e6312f9efc8c67c"} err="failed to get container status \"e76febed948f35df2fda7564f886a26f4232d64aee5f448e4e6312f9efc8c67c\": rpc error: code = NotFound desc = could not find container \"e76febed948f35df2fda7564f886a26f4232d64aee5f448e4e6312f9efc8c67c\": container with ID starting with e76febed948f35df2fda7564f886a26f4232d64aee5f448e4e6312f9efc8c67c not found: ID does not exist"
Aug 04 18:26:31.771209 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:26:31.771167    2583 scope.go:117] "RemoveContainer" containerID="fd6a4fbbe13fc1529a4de83891b7b5cce1c3e3b21e9edb95475c8867d694481f"
Aug 04 18:26:31.771404 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:26:31.771387    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd6a4fbbe13fc1529a4de83891b7b5cce1c3e3b21e9edb95475c8867d694481f\": container with ID starting with fd6a4fbbe13fc1529a4de83891b7b5cce1c3e3b21e9edb95475c8867d694481f not found: ID does not exist" containerID="fd6a4fbbe13fc1529a4de83891b7b5cce1c3e3b21e9edb95475c8867d694481f"
Aug 04 18:26:31.771451 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:26:31.771410    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd6a4fbbe13fc1529a4de83891b7b5cce1c3e3b21e9edb95475c8867d694481f"} err="failed to get container status \"fd6a4fbbe13fc1529a4de83891b7b5cce1c3e3b21e9edb95475c8867d694481f\": rpc error: code = NotFound desc = could not find container \"fd6a4fbbe13fc1529a4de83891b7b5cce1c3e3b21e9edb95475c8867d694481f\": container with ID starting with fd6a4fbbe13fc1529a4de83891b7b5cce1c3e3b21e9edb95475c8867d694481f not found: ID does not exist"
Aug 04 18:26:31.782741 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:26:31.782722    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-56649fd646-mwqdj"]
Aug 04 18:26:31.787459 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:26:31.787434    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-56649fd646-mwqdj"]
Aug 04 18:26:31.797595 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:26:31.797580    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18fc9d06-f3fa-4437-b889-a9e2b2fb767d-tls-certs\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:26:31.797595 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:26:31.797597    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18fc9d06-f3fa-4437-b889-a9e2b2fb767d-tmp-dir\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:26:31.797698 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:26:31.797606    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/18fc9d06-f3fa-4437-b889-a9e2b2fb767d-kserve-provision-location\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:26:31.797698 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:26:31.797616    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/18fc9d06-f3fa-4437-b889-a9e2b2fb767d-cabundle-cert\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:26:32.326553 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:26:32.326514    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-56649fd646-mwqdj" podUID="83d92453-9393-44cc-885a-403f1c5a6ba5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Aug 04 18:26:32.942117 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:26:32.942084    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="18fc9d06-f3fa-4437-b889-a9e2b2fb767d" path="/var/lib/kubelet/pods/18fc9d06-f3fa-4437-b889-a9e2b2fb767d/volumes"
Aug 04 18:26:32.942589 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:26:32.942576    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="83d92453-9393-44cc-885a-403f1c5a6ba5" path="/var/lib/kubelet/pods/83d92453-9393-44cc-885a-403f1c5a6ba5/volumes"
Aug 04 18:27:00.828684 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:27:00.828651    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9d7889df5-6c2xx"]
Aug 04 18:27:00.829194 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:27:00.829011    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="18fc9d06-f3fa-4437-b889-a9e2b2fb767d" containerName="main"
Aug 04 18:27:00.829194 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:27:00.829022    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="18fc9d06-f3fa-4437-b889-a9e2b2fb767d" containerName="main"
Aug 04 18:27:00.829194 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:27:00.829042    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83d92453-9393-44cc-885a-403f1c5a6ba5" containerName="storage-initializer"
Aug 04 18:27:00.829194 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:27:00.829049    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="83d92453-9393-44cc-885a-403f1c5a6ba5" containerName="storage-initializer"
Aug 04 18:27:00.829194 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:27:00.829057    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="18fc9d06-f3fa-4437-b889-a9e2b2fb767d" containerName="storage-initializer"
Aug 04 18:27:00.829194 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:27:00.829062    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="18fc9d06-f3fa-4437-b889-a9e2b2fb767d" containerName="storage-initializer"
Aug 04 18:27:00.829194 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:27:00.829070    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83d92453-9393-44cc-885a-403f1c5a6ba5" containerName="main"
Aug 04 18:27:00.829194 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:27:00.829075    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="83d92453-9393-44cc-885a-403f1c5a6ba5" containerName="main"
Aug 04 18:27:00.829194 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:27:00.829129    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="18fc9d06-f3fa-4437-b889-a9e2b2fb767d" containerName="main"
Aug 04 18:27:00.829194 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:27:00.829135    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="83d92453-9393-44cc-885a-403f1c5a6ba5" containerName="main"
Aug 04 18:27:00.832320 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:27:00.832299    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9d7889df5-6c2xx"
Aug 04 18:27:00.835162 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:27:00.835139    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"openshift-service-ca.crt\""
Aug 04 18:27:00.835297 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:27:00.835175    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"default-dockercfg-vr6g7\""
Aug 04 18:27:00.836237 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:27:00.836216    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"seaweedfs-s3-creds\""
Aug 04 18:27:00.836237 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:27:00.836228    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"kube-root-ca.crt\""
Aug 04 18:27:00.836368 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:27:00.836248    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"odh-kserve-custom-ca-bundle\""
Aug 04 18:27:00.836368 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:27:00.836257    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"tokenizer-migration-path-test-kserve-self-signed-certs\""
Aug 04 18:27:00.844276 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:27:00.844252    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9d7889df5-6c2xx"]
Aug 04 18:27:00.942293 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:27:00.942261    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfhqg\" (UniqueName: \"kubernetes.io/projected/41c42cfc-2480-4bb7-9c60-7da3336dab9f-kube-api-access-mfhqg\") pod \"tokenizer-migration-path-test-kserve-9d7889df5-6c2xx\" (UID: \"41c42cfc-2480-4bb7-9c60-7da3336dab9f\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9d7889df5-6c2xx"
Aug 04 18:27:00.942498 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:27:00.942318    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/41c42cfc-2480-4bb7-9c60-7da3336dab9f-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-9d7889df5-6c2xx\" (UID: \"41c42cfc-2480-4bb7-9c60-7da3336dab9f\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9d7889df5-6c2xx"
Aug 04 18:27:00.942498 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:27:00.942348    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/41c42cfc-2480-4bb7-9c60-7da3336dab9f-model-cache\") pod \"tokenizer-migration-path-test-kserve-9d7889df5-6c2xx\" (UID: \"41c42cfc-2480-4bb7-9c60-7da3336dab9f\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9d7889df5-6c2xx"
Aug 04 18:27:00.942498 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:27:00.942387    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/41c42cfc-2480-4bb7-9c60-7da3336dab9f-home\") pod \"tokenizer-migration-path-test-kserve-9d7889df5-6c2xx\" (UID: \"41c42cfc-2480-4bb7-9c60-7da3336dab9f\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9d7889df5-6c2xx"
Aug 04 18:27:00.942498 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:27:00.942407    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41c42cfc-2480-4bb7-9c60-7da3336dab9f-tls-certs\") pod \"tokenizer-migration-path-test-kserve-9d7889df5-6c2xx\" (UID: \"41c42cfc-2480-4bb7-9c60-7da3336dab9f\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9d7889df5-6c2xx"
Aug 04 18:27:00.942679 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:27:00.942494    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41c42cfc-2480-4bb7-9c60-7da3336dab9f-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-9d7889df5-6c2xx\" (UID: \"41c42cfc-2480-4bb7-9c60-7da3336dab9f\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9d7889df5-6c2xx"
Aug 04 18:27:00.942679 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:27:00.942521    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/41c42cfc-2480-4bb7-9c60-7da3336dab9f-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-9d7889df5-6c2xx\" (UID: \"41c42cfc-2480-4bb7-9c60-7da3336dab9f\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9d7889df5-6c2xx"
Aug 04 18:27:00.942679 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:27:00.942549    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/41c42cfc-2480-4bb7-9c60-7da3336dab9f-dshm\") pod \"tokenizer-migration-path-test-kserve-9d7889df5-6c2xx\" (UID: \"41c42cfc-2480-4bb7-9c60-7da3336dab9f\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9d7889df5-6c2xx"
Aug 04 18:27:01.043134 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:27:01.043099    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41c42cfc-2480-4bb7-9c60-7da3336dab9f-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-9d7889df5-6c2xx\" (UID: \"41c42cfc-2480-4bb7-9c60-7da3336dab9f\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9d7889df5-6c2xx"
Aug 04 18:27:01.043318 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:27:01.043143    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/41c42cfc-2480-4bb7-9c60-7da3336dab9f-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-9d7889df5-6c2xx\" (UID: \"41c42cfc-2480-4bb7-9c60-7da3336dab9f\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9d7889df5-6c2xx"
Aug 04 18:27:01.043318 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:27:01.043167    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/41c42cfc-2480-4bb7-9c60-7da3336dab9f-dshm\") pod \"tokenizer-migration-path-test-kserve-9d7889df5-6c2xx\" (UID: \"41c42cfc-2480-4bb7-9c60-7da3336dab9f\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9d7889df5-6c2xx"
Aug 04 18:27:01.043318 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:27:01.043196    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfhqg\" (UniqueName: \"kubernetes.io/projected/41c42cfc-2480-4bb7-9c60-7da3336dab9f-kube-api-access-mfhqg\") pod \"tokenizer-migration-path-test-kserve-9d7889df5-6c2xx\" (UID: \"41c42cfc-2480-4bb7-9c60-7da3336dab9f\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9d7889df5-6c2xx"
Aug 04 18:27:01.043523 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:27:01.043357    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/41c42cfc-2480-4bb7-9c60-7da3336dab9f-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-9d7889df5-6c2xx\" (UID: \"41c42cfc-2480-4bb7-9c60-7da3336dab9f\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9d7889df5-6c2xx"
Aug 04 18:27:01.043523 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:27:01.043395    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/41c42cfc-2480-4bb7-9c60-7da3336dab9f-model-cache\") pod \"tokenizer-migration-path-test-kserve-9d7889df5-6c2xx\" (UID: \"41c42cfc-2480-4bb7-9c60-7da3336dab9f\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9d7889df5-6c2xx"
Aug 04 18:27:01.043523 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:27:01.043444    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/41c42cfc-2480-4bb7-9c60-7da3336dab9f-home\") pod \"tokenizer-migration-path-test-kserve-9d7889df5-6c2xx\" (UID: \"41c42cfc-2480-4bb7-9c60-7da3336dab9f\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9d7889df5-6c2xx"
Aug 04 18:27:01.043523 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:27:01.043490    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41c42cfc-2480-4bb7-9c60-7da3336dab9f-tls-certs\") pod \"tokenizer-migration-path-test-kserve-9d7889df5-6c2xx\" (UID: \"41c42cfc-2480-4bb7-9c60-7da3336dab9f\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9d7889df5-6c2xx"
Aug 04 18:27:01.043757 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:27:01.043605    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41c42cfc-2480-4bb7-9c60-7da3336dab9f-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-9d7889df5-6c2xx\" (UID: \"41c42cfc-2480-4bb7-9c60-7da3336dab9f\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9d7889df5-6c2xx"
Aug 04 18:27:01.044063 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:27:01.044035    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/41c42cfc-2480-4bb7-9c60-7da3336dab9f-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-9d7889df5-6c2xx\" (UID: \"41c42cfc-2480-4bb7-9c60-7da3336dab9f\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9d7889df5-6c2xx"
Aug 04 18:27:01.044225 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:27:01.044133    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/41c42cfc-2480-4bb7-9c60-7da3336dab9f-model-cache\") pod \"tokenizer-migration-path-test-kserve-9d7889df5-6c2xx\" (UID: \"41c42cfc-2480-4bb7-9c60-7da3336dab9f\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9d7889df5-6c2xx"
Aug 04 18:27:01.044330 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:27:01.044229    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/41c42cfc-2480-4bb7-9c60-7da3336dab9f-home\") pod \"tokenizer-migration-path-test-kserve-9d7889df5-6c2xx\" (UID: \"41c42cfc-2480-4bb7-9c60-7da3336dab9f\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9d7889df5-6c2xx"
Aug 04 18:27:01.044559 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:27:01.044425    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/41c42cfc-2480-4bb7-9c60-7da3336dab9f-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-9d7889df5-6c2xx\" (UID: \"41c42cfc-2480-4bb7-9c60-7da3336dab9f\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9d7889df5-6c2xx"
Aug 04 18:27:01.046072 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:27:01.046049    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/41c42cfc-2480-4bb7-9c60-7da3336dab9f-dshm\") pod \"tokenizer-migration-path-test-kserve-9d7889df5-6c2xx\" (UID: \"41c42cfc-2480-4bb7-9c60-7da3336dab9f\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9d7889df5-6c2xx"
Aug 04 18:27:01.046335 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:27:01.046317    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41c42cfc-2480-4bb7-9c60-7da3336dab9f-tls-certs\") pod \"tokenizer-migration-path-test-kserve-9d7889df5-6c2xx\" (UID: \"41c42cfc-2480-4bb7-9c60-7da3336dab9f\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9d7889df5-6c2xx"
Aug 04 18:27:01.054085 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:27:01.054036    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfhqg\" (UniqueName: \"kubernetes.io/projected/41c42cfc-2480-4bb7-9c60-7da3336dab9f-kube-api-access-mfhqg\") pod \"tokenizer-migration-path-test-kserve-9d7889df5-6c2xx\" (UID: \"41c42cfc-2480-4bb7-9c60-7da3336dab9f\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9d7889df5-6c2xx"
Aug 04 18:27:01.057071 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:27:01.057048    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-6b68df8465-mv8ft"]
Aug 04 18:27:01.062537 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:27:01.062519    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-6b68df8465-mv8ft"
Aug 04 18:27:01.081278 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:27:01.081212    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-6b68df8465-mv8ft"]
Aug 04 18:27:01.144555 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:27:01.144517    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9d7889df5-6c2xx"
Aug 04 18:27:01.144730 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:27:01.144703    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/083f6773-b9f4-4bcf-894a-9eb325e0c4d9-cabundle-cert\") pod \"tokenizer-migration-path-test-tokenizer-6b68df8465-mv8ft\" (UID: \"083f6773-b9f4-4bcf-894a-9eb325e0c4d9\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-6b68df8465-mv8ft"
Aug 04 18:27:01.144796 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:27:01.144758    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/083f6773-b9f4-4bcf-894a-9eb325e0c4d9-tmp-dir\") pod \"tokenizer-migration-path-test-tokenizer-6b68df8465-mv8ft\" (UID: \"083f6773-b9f4-4bcf-894a-9eb325e0c4d9\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-6b68df8465-mv8ft"
Aug 04 18:27:01.144796 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:27:01.144778    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/083f6773-b9f4-4bcf-894a-9eb325e0c4d9-kserve-provision-location\") pod \"tokenizer-migration-path-test-tokenizer-6b68df8465-mv8ft\" (UID: \"083f6773-b9f4-4bcf-894a-9eb325e0c4d9\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-6b68df8465-mv8ft"
Aug 04 18:27:01.144946 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:27:01.144923    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/083f6773-b9f4-4bcf-894a-9eb325e0c4d9-tls-certs\") pod \"tokenizer-migration-path-test-tokenizer-6b68df8465-mv8ft\" (UID: \"083f6773-b9f4-4bcf-894a-9eb325e0c4d9\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-6b68df8465-mv8ft"
Aug 04 18:27:01.246201 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:27:01.246164    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/083f6773-b9f4-4bcf-894a-9eb325e0c4d9-tls-certs\") pod \"tokenizer-migration-path-test-tokenizer-6b68df8465-mv8ft\" (UID: \"083f6773-b9f4-4bcf-894a-9eb325e0c4d9\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-6b68df8465-mv8ft"
Aug 04 18:27:01.246364 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:27:01.246217    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/083f6773-b9f4-4bcf-894a-9eb325e0c4d9-cabundle-cert\") pod \"tokenizer-migration-path-test-tokenizer-6b68df8465-mv8ft\" (UID: \"083f6773-b9f4-4bcf-894a-9eb325e0c4d9\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-6b68df8465-mv8ft"
Aug 04 18:27:01.246364 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:27:01.246273    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/083f6773-b9f4-4bcf-894a-9eb325e0c4d9-tmp-dir\") pod \"tokenizer-migration-path-test-tokenizer-6b68df8465-mv8ft\" (UID: \"083f6773-b9f4-4bcf-894a-9eb325e0c4d9\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-6b68df8465-mv8ft"
Aug 04 18:27:01.246364 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:27:01.246302    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/083f6773-b9f4-4bcf-894a-9eb325e0c4d9-kserve-provision-location\") pod \"tokenizer-migration-path-test-tokenizer-6b68df8465-mv8ft\" (UID: \"083f6773-b9f4-4bcf-894a-9eb325e0c4d9\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-6b68df8465-mv8ft"
Aug 04 18:27:01.246775 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:27:01.246747    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/083f6773-b9f4-4bcf-894a-9eb325e0c4d9-tmp-dir\") pod \"tokenizer-migration-path-test-tokenizer-6b68df8465-mv8ft\" (UID: \"083f6773-b9f4-4bcf-894a-9eb325e0c4d9\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-6b68df8465-mv8ft"
Aug 04 18:27:01.246997 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:27:01.246941    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/083f6773-b9f4-4bcf-894a-9eb325e0c4d9-kserve-provision-location\") pod \"tokenizer-migration-path-test-tokenizer-6b68df8465-mv8ft\" (UID: \"083f6773-b9f4-4bcf-894a-9eb325e0c4d9\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-6b68df8465-mv8ft"
Aug 04 18:27:01.247351 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:27:01.247323    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/083f6773-b9f4-4bcf-894a-9eb325e0c4d9-cabundle-cert\") pod \"tokenizer-migration-path-test-tokenizer-6b68df8465-mv8ft\" (UID: \"083f6773-b9f4-4bcf-894a-9eb325e0c4d9\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-6b68df8465-mv8ft"
Aug 04 18:27:01.248784 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:27:01.248762    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/083f6773-b9f4-4bcf-894a-9eb325e0c4d9-tls-certs\") pod \"tokenizer-migration-path-test-tokenizer-6b68df8465-mv8ft\" (UID: \"083f6773-b9f4-4bcf-894a-9eb325e0c4d9\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-6b68df8465-mv8ft"
Aug 04 18:27:01.286665 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:27:01.286640    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9d7889df5-6c2xx"]
Aug 04 18:27:01.288074 ip-10-0-135-124 kubenswrapper[2583]: W0804 18:27:01.288047    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41c42cfc_2480_4bb7_9c60_7da3336dab9f.slice/crio-a7a7fb3498999cd3ab1d7ffcc67a5a1553a26a308ef84b51a4bb52b7fcbf896e WatchSource:0}: Error finding container a7a7fb3498999cd3ab1d7ffcc67a5a1553a26a308ef84b51a4bb52b7fcbf896e: Status 404 returned error can't find the container with id a7a7fb3498999cd3ab1d7ffcc67a5a1553a26a308ef84b51a4bb52b7fcbf896e
Aug 04 18:27:01.386641 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:27:01.386615    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-6b68df8465-mv8ft"
Aug 04 18:27:01.530842 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:27:01.530817    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-6b68df8465-mv8ft"]
Aug 04 18:27:01.533538 ip-10-0-135-124 kubenswrapper[2583]: W0804 18:27:01.533507    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod083f6773_b9f4_4bcf_894a_9eb325e0c4d9.slice/crio-f2f7e8f2f44180d0d4b14d64eea19050f8e7844c8207c9ab750e5cc9fa4fbb91 WatchSource:0}: Error finding container f2f7e8f2f44180d0d4b14d64eea19050f8e7844c8207c9ab750e5cc9fa4fbb91: Status 404 returned error can't find the container with id f2f7e8f2f44180d0d4b14d64eea19050f8e7844c8207c9ab750e5cc9fa4fbb91
Aug 04 18:27:01.901735 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:27:01.901637    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9d7889df5-6c2xx" event={"ID":"41c42cfc-2480-4bb7-9c60-7da3336dab9f","Type":"ContainerStarted","Data":"73b193b7971488a45dd2c63bd4363a921b78549f222dc5f53d49851ae8fd41be"}
Aug 04 18:27:01.901735 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:27:01.901711    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9d7889df5-6c2xx" event={"ID":"41c42cfc-2480-4bb7-9c60-7da3336dab9f","Type":"ContainerStarted","Data":"a7a7fb3498999cd3ab1d7ffcc67a5a1553a26a308ef84b51a4bb52b7fcbf896e"}
Aug 04 18:27:01.903362 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:27:01.903328    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-6b68df8465-mv8ft" event={"ID":"083f6773-b9f4-4bcf-894a-9eb325e0c4d9","Type":"ContainerStarted","Data":"aeabdf54cdc44e12dc3ddebd5b34fff5e77ff44ab34665c185ccb9dc20f897a3"}
Aug 04 18:27:01.903535 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:27:01.903373    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-6b68df8465-mv8ft" event={"ID":"083f6773-b9f4-4bcf-894a-9eb325e0c4d9","Type":"ContainerStarted","Data":"f2f7e8f2f44180d0d4b14d64eea19050f8e7844c8207c9ab750e5cc9fa4fbb91"}
Aug 04 18:27:03.917398 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:27:03.917369    2583 generic.go:356] "Generic (PLEG): container finished" podID="41c42cfc-2480-4bb7-9c60-7da3336dab9f" containerID="73b193b7971488a45dd2c63bd4363a921b78549f222dc5f53d49851ae8fd41be" exitCode=0
Aug 04 18:27:03.917855 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:27:03.917438    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9d7889df5-6c2xx" event={"ID":"41c42cfc-2480-4bb7-9c60-7da3336dab9f","Type":"ContainerDied","Data":"73b193b7971488a45dd2c63bd4363a921b78549f222dc5f53d49851ae8fd41be"}
Aug 04 18:27:04.925079 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:27:04.925045    2583 generic.go:356] "Generic (PLEG): container finished" podID="083f6773-b9f4-4bcf-894a-9eb325e0c4d9" containerID="aeabdf54cdc44e12dc3ddebd5b34fff5e77ff44ab34665c185ccb9dc20f897a3" exitCode=0
Aug 04 18:27:04.925546 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:27:04.925126    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-6b68df8465-mv8ft" event={"ID":"083f6773-b9f4-4bcf-894a-9eb325e0c4d9","Type":"ContainerDied","Data":"aeabdf54cdc44e12dc3ddebd5b34fff5e77ff44ab34665c185ccb9dc20f897a3"}
Aug 04 18:27:04.927312 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:27:04.927289    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9d7889df5-6c2xx" event={"ID":"41c42cfc-2480-4bb7-9c60-7da3336dab9f","Type":"ContainerStarted","Data":"77a6c198f9322eb992eff4f910e15baeab7f437291d3fef21af3c1ac6357275b"}
Aug 04 18:27:04.967035 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:27:04.966994    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9d7889df5-6c2xx" podStartSLOduration=2.576906383 podStartE2EDuration="4.966979105s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.390072722s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:27:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:27:04.964696249 +0000 UTC m=+1638.545914827" watchObservedRunningTime="2026-08-04 18:27:04.966979105 +0000 UTC m=+1638.548197682"
Aug 04 18:27:05.145457 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:27:05.145424    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9d7889df5-6c2xx"
Aug 04 18:27:05.935087 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:27:05.935042    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-6b68df8465-mv8ft" event={"ID":"083f6773-b9f4-4bcf-894a-9eb325e0c4d9","Type":"ContainerStarted","Data":"327da06461652632319b64ad73178fac702dbd31746828cd63ee9df185268e84"}
Aug 04 18:27:05.935499 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:27:05.935401    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-6b68df8465-mv8ft"
Aug 04 18:27:05.957311 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:27:05.957254    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-6b68df8465-mv8ft" podStartSLOduration=2.527787567 podStartE2EDuration="4.95723444s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.429446873s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:27:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:27:05.954238697 +0000 UTC m=+1639.535457275" watchObservedRunningTime="2026-08-04 18:27:05.95723444 +0000 UTC m=+1639.538453020"
Aug 04 18:27:11.145310 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:27:11.145258    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9d7889df5-6c2xx"
Aug 04 18:27:11.158056 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:27:11.158031    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9d7889df5-6c2xx"
Aug 04 18:27:11.989939 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:27:11.989907    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9d7889df5-6c2xx"
Aug 04 18:27:13.005546 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:27:13.005512    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-xbsx9"]
Aug 04 18:27:28.026116 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:27:28.026051    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-xbsx9" podUID="0bd3381e-6544-4ad5-86ad-2bdf37e2558c" containerName="main" containerID="cri-o://0889f737a66a8498d2eb966872a511b7886302565efd4e302c500f4acc9299cf" gracePeriod=45
Aug 04 18:27:28.298554 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:27:28.298530    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-xbsx9"
Aug 04 18:27:28.397882 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:27:28.397851    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jwvsq\" (UniqueName: \"kubernetes.io/projected/0bd3381e-6544-4ad5-86ad-2bdf37e2558c-kube-api-access-jwvsq\") pod \"0bd3381e-6544-4ad5-86ad-2bdf37e2558c\" (UID: \"0bd3381e-6544-4ad5-86ad-2bdf37e2558c\") "
Aug 04 18:27:28.397882 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:27:28.397887    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0bd3381e-6544-4ad5-86ad-2bdf37e2558c-model-cache\") pod \"0bd3381e-6544-4ad5-86ad-2bdf37e2558c\" (UID: \"0bd3381e-6544-4ad5-86ad-2bdf37e2558c\") "
Aug 04 18:27:28.398095 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:27:28.397936    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0bd3381e-6544-4ad5-86ad-2bdf37e2558c-home\") pod \"0bd3381e-6544-4ad5-86ad-2bdf37e2558c\" (UID: \"0bd3381e-6544-4ad5-86ad-2bdf37e2558c\") "
Aug 04 18:27:28.398095 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:27:28.397966    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0bd3381e-6544-4ad5-86ad-2bdf37e2558c-tls-certs\") pod \"0bd3381e-6544-4ad5-86ad-2bdf37e2558c\" (UID: \"0bd3381e-6544-4ad5-86ad-2bdf37e2558c\") "
Aug 04 18:27:28.398095 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:27:28.397989    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0bd3381e-6544-4ad5-86ad-2bdf37e2558c-dshm\") pod \"0bd3381e-6544-4ad5-86ad-2bdf37e2558c\" (UID: \"0bd3381e-6544-4ad5-86ad-2bdf37e2558c\") "
Aug 04 18:27:28.398095 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:27:28.398013    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0bd3381e-6544-4ad5-86ad-2bdf37e2558c-tmp-dir\") pod \"0bd3381e-6544-4ad5-86ad-2bdf37e2558c\" (UID: \"0bd3381e-6544-4ad5-86ad-2bdf37e2558c\") "
Aug 04 18:27:28.398299 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:27:28.398157    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bd3381e-6544-4ad5-86ad-2bdf37e2558c-model-cache" (OuterVolumeSpecName: "model-cache") pod "0bd3381e-6544-4ad5-86ad-2bdf37e2558c" (UID: "0bd3381e-6544-4ad5-86ad-2bdf37e2558c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:27:28.398299 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:27:28.398242    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bd3381e-6544-4ad5-86ad-2bdf37e2558c-home" (OuterVolumeSpecName: "home") pod "0bd3381e-6544-4ad5-86ad-2bdf37e2558c" (UID: "0bd3381e-6544-4ad5-86ad-2bdf37e2558c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:27:28.398419 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:27:28.398319    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0bd3381e-6544-4ad5-86ad-2bdf37e2558c-model-cache\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:27:28.398419 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:27:28.398325    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bd3381e-6544-4ad5-86ad-2bdf37e2558c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0bd3381e-6544-4ad5-86ad-2bdf37e2558c" (UID: "0bd3381e-6544-4ad5-86ad-2bdf37e2558c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:27:28.400167 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:27:28.400134    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bd3381e-6544-4ad5-86ad-2bdf37e2558c-dshm" (OuterVolumeSpecName: "dshm") pod "0bd3381e-6544-4ad5-86ad-2bdf37e2558c" (UID: "0bd3381e-6544-4ad5-86ad-2bdf37e2558c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:27:28.400167 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:27:28.400156    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bd3381e-6544-4ad5-86ad-2bdf37e2558c-kube-api-access-jwvsq" (OuterVolumeSpecName: "kube-api-access-jwvsq") pod "0bd3381e-6544-4ad5-86ad-2bdf37e2558c" (UID: "0bd3381e-6544-4ad5-86ad-2bdf37e2558c"). InnerVolumeSpecName "kube-api-access-jwvsq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:27:28.400330 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:27:28.400185    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0bd3381e-6544-4ad5-86ad-2bdf37e2558c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0bd3381e-6544-4ad5-86ad-2bdf37e2558c" (UID: "0bd3381e-6544-4ad5-86ad-2bdf37e2558c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:27:28.498858 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:27:28.498827    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0bd3381e-6544-4ad5-86ad-2bdf37e2558c-home\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:27:28.498858 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:27:28.498853    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0bd3381e-6544-4ad5-86ad-2bdf37e2558c-tls-certs\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:27:28.498858 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:27:28.498863    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0bd3381e-6544-4ad5-86ad-2bdf37e2558c-dshm\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:27:28.499059 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:27:28.498871    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0bd3381e-6544-4ad5-86ad-2bdf37e2558c-tmp-dir\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:27:28.499059 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:27:28.498879    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jwvsq\" (UniqueName: \"kubernetes.io/projected/0bd3381e-6544-4ad5-86ad-2bdf37e2558c-kube-api-access-jwvsq\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:27:29.081280 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:27:29.081252    2583 generic.go:356] "Generic (PLEG): container finished" podID="0bd3381e-6544-4ad5-86ad-2bdf37e2558c" containerID="0889f737a66a8498d2eb966872a511b7886302565efd4e302c500f4acc9299cf" exitCode=0
Aug 04 18:27:29.081753 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:27:29.081310    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-xbsx9"
Aug 04 18:27:29.081753 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:27:29.081343    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-xbsx9" event={"ID":"0bd3381e-6544-4ad5-86ad-2bdf37e2558c","Type":"ContainerDied","Data":"0889f737a66a8498d2eb966872a511b7886302565efd4e302c500f4acc9299cf"}
Aug 04 18:27:29.081753 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:27:29.081415    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-xbsx9" event={"ID":"0bd3381e-6544-4ad5-86ad-2bdf37e2558c","Type":"ContainerDied","Data":"00ed63a5a7f15659040747843587ec7d6d2f45df5a833ccf6499913bcd654cf6"}
Aug 04 18:27:29.081753 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:27:29.081449    2583 scope.go:117] "RemoveContainer" containerID="0889f737a66a8498d2eb966872a511b7886302565efd4e302c500f4acc9299cf"
Aug 04 18:27:29.091028 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:27:29.091012    2583 scope.go:117] "RemoveContainer" containerID="0889f737a66a8498d2eb966872a511b7886302565efd4e302c500f4acc9299cf"
Aug 04 18:27:29.091300 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:27:29.091281    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0889f737a66a8498d2eb966872a511b7886302565efd4e302c500f4acc9299cf\": container with ID starting with 0889f737a66a8498d2eb966872a511b7886302565efd4e302c500f4acc9299cf not found: ID does not exist" containerID="0889f737a66a8498d2eb966872a511b7886302565efd4e302c500f4acc9299cf"
Aug 04 18:27:29.091362 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:27:29.091313    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0889f737a66a8498d2eb966872a511b7886302565efd4e302c500f4acc9299cf"} err="failed to get container status \"0889f737a66a8498d2eb966872a511b7886302565efd4e302c500f4acc9299cf\": rpc error: code = NotFound desc = could not find container \"0889f737a66a8498d2eb966872a511b7886302565efd4e302c500f4acc9299cf\": container with ID starting with 0889f737a66a8498d2eb966872a511b7886302565efd4e302c500f4acc9299cf not found: ID does not exist"
Aug 04 18:27:29.099717 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:27:29.099694    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-xbsx9"]
Aug 04 18:27:29.105209 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:27:29.105185    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-xbsx9"]
Aug 04 18:27:30.943136 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:27:30.943106    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0bd3381e-6544-4ad5-86ad-2bdf37e2558c" path="/var/lib/kubelet/pods/0bd3381e-6544-4ad5-86ad-2bdf37e2558c/volumes"
Aug 04 18:27:36.955232 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:27:36.955199    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-6b68df8465-mv8ft"
Aug 04 18:27:56.051604 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:27:56.051570    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-312caf2d_tokenizer-migration-path-test-tokenizer-6b68df8465-mv8ft_083f6773-b9f4-4bcf-894a-9eb325e0c4d9/main/0.log"
Aug 04 18:27:56.565684 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:27:56.565645    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-6b68df8465-mv8ft"]
Aug 04 18:27:56.566974 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:27:56.566939    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-6b68df8465-mv8ft" podUID="083f6773-b9f4-4bcf-894a-9eb325e0c4d9" containerName="main" containerID="cri-o://327da06461652632319b64ad73178fac702dbd31746828cd63ee9df185268e84" gracePeriod=30
Aug 04 18:27:56.569646 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:27:56.569616    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9d7889df5-6c2xx"]
Aug 04 18:27:56.578700 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:27:56.578660    2583 log.go:32] "ExecSync cmd from runtime service failed" err=<
Aug 04 18:27:56.578700 ip-10-0-135-124 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Aug 04 18:27:56.578700 ip-10-0-135-124 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Aug 04 18:27:56.578700 ip-10-0-135-124 kubenswrapper[2583]:  > containerID="77a6c198f9322eb992eff4f910e15baeab7f437291d3fef21af3c1ac6357275b" cmd=["/bin/sleep","15"]
Aug 04 18:27:56.578922 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:27:56.578719    2583 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9d7889df5-6c2xx" message="" err=<
Aug 04 18:27:56.578922 ip-10-0-135-124 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Aug 04 18:27:56.578922 ip-10-0-135-124 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Aug 04 18:27:56.578922 ip-10-0-135-124 kubenswrapper[2583]:  >
Aug 04 18:27:56.578922 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:27:56.578733    2583 kuberuntime_container.go:755] "PreStop hook failed" err=<
Aug 04 18:27:56.578922 ip-10-0-135-124 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Aug 04 18:27:56.578922 ip-10-0-135-124 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Aug 04 18:27:56.578922 ip-10-0-135-124 kubenswrapper[2583]:  > pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9d7889df5-6c2xx" podUID="41c42cfc-2480-4bb7-9c60-7da3336dab9f" containerName="main" containerID="cri-o://77a6c198f9322eb992eff4f910e15baeab7f437291d3fef21af3c1ac6357275b"
Aug 04 18:27:56.578922 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:27:56.578780    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9d7889df5-6c2xx" podUID="41c42cfc-2480-4bb7-9c60-7da3336dab9f" containerName="main" containerID="cri-o://77a6c198f9322eb992eff4f910e15baeab7f437291d3fef21af3c1ac6357275b" gracePeriod=60
Aug 04 18:27:56.934926 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:27:56.934897    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9d7889df5-6c2xx"
Aug 04 18:27:56.943930 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:27:56.943888    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-6b68df8465-mv8ft" podUID="083f6773-b9f4-4bcf-894a-9eb325e0c4d9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": dial tcp 10.132.0.62:8000: connect: connection refused"
Aug 04 18:27:57.022906 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:27:57.022870    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mfhqg\" (UniqueName: \"kubernetes.io/projected/41c42cfc-2480-4bb7-9c60-7da3336dab9f-kube-api-access-mfhqg\") pod \"41c42cfc-2480-4bb7-9c60-7da3336dab9f\" (UID: \"41c42cfc-2480-4bb7-9c60-7da3336dab9f\") "
Aug 04 18:27:57.023066 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:27:57.022909    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41c42cfc-2480-4bb7-9c60-7da3336dab9f-tls-certs\") pod \"41c42cfc-2480-4bb7-9c60-7da3336dab9f\" (UID: \"41c42cfc-2480-4bb7-9c60-7da3336dab9f\") "
Aug 04 18:27:57.023066 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:27:57.023001    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/41c42cfc-2480-4bb7-9c60-7da3336dab9f-cabundle-cert\") pod \"41c42cfc-2480-4bb7-9c60-7da3336dab9f\" (UID: \"41c42cfc-2480-4bb7-9c60-7da3336dab9f\") "
Aug 04 18:27:57.023066 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:27:57.023026    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/41c42cfc-2480-4bb7-9c60-7da3336dab9f-home\") pod \"41c42cfc-2480-4bb7-9c60-7da3336dab9f\" (UID: \"41c42cfc-2480-4bb7-9c60-7da3336dab9f\") "
Aug 04 18:27:57.023066 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:27:57.023062    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41c42cfc-2480-4bb7-9c60-7da3336dab9f-tmp-dir\") pod \"41c42cfc-2480-4bb7-9c60-7da3336dab9f\" (UID: \"41c42cfc-2480-4bb7-9c60-7da3336dab9f\") "
Aug 04 18:27:57.023288 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:27:57.023106    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/41c42cfc-2480-4bb7-9c60-7da3336dab9f-kserve-provision-location\") pod \"41c42cfc-2480-4bb7-9c60-7da3336dab9f\" (UID: \"41c42cfc-2480-4bb7-9c60-7da3336dab9f\") "
Aug 04 18:27:57.023288 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:27:57.023133    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/41c42cfc-2480-4bb7-9c60-7da3336dab9f-model-cache\") pod \"41c42cfc-2480-4bb7-9c60-7da3336dab9f\" (UID: \"41c42cfc-2480-4bb7-9c60-7da3336dab9f\") "
Aug 04 18:27:57.023288 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:27:57.023156    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/41c42cfc-2480-4bb7-9c60-7da3336dab9f-dshm\") pod \"41c42cfc-2480-4bb7-9c60-7da3336dab9f\" (UID: \"41c42cfc-2480-4bb7-9c60-7da3336dab9f\") "
Aug 04 18:27:57.023448 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:27:57.023318    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41c42cfc-2480-4bb7-9c60-7da3336dab9f-home" (OuterVolumeSpecName: "home") pod "41c42cfc-2480-4bb7-9c60-7da3336dab9f" (UID: "41c42cfc-2480-4bb7-9c60-7da3336dab9f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:27:57.023448 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:27:57.023343    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41c42cfc-2480-4bb7-9c60-7da3336dab9f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "41c42cfc-2480-4bb7-9c60-7da3336dab9f" (UID: "41c42cfc-2480-4bb7-9c60-7da3336dab9f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:27:57.023581 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:27:57.023511    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41c42cfc-2480-4bb7-9c60-7da3336dab9f-model-cache" (OuterVolumeSpecName: "model-cache") pod "41c42cfc-2480-4bb7-9c60-7da3336dab9f" (UID: "41c42cfc-2480-4bb7-9c60-7da3336dab9f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:27:57.023581 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:27:57.023521    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41c42cfc-2480-4bb7-9c60-7da3336dab9f-tmp-dir\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:27:57.023581 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:27:57.023538    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/41c42cfc-2480-4bb7-9c60-7da3336dab9f-home\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:27:57.023581 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:27:57.023393    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41c42cfc-2480-4bb7-9c60-7da3336dab9f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "41c42cfc-2480-4bb7-9c60-7da3336dab9f" (UID: "41c42cfc-2480-4bb7-9c60-7da3336dab9f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:27:57.025400 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:27:57.025370    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41c42cfc-2480-4bb7-9c60-7da3336dab9f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "41c42cfc-2480-4bb7-9c60-7da3336dab9f" (UID: "41c42cfc-2480-4bb7-9c60-7da3336dab9f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:27:57.025400 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:27:57.025378    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41c42cfc-2480-4bb7-9c60-7da3336dab9f-kube-api-access-mfhqg" (OuterVolumeSpecName: "kube-api-access-mfhqg") pod "41c42cfc-2480-4bb7-9c60-7da3336dab9f" (UID: "41c42cfc-2480-4bb7-9c60-7da3336dab9f"). InnerVolumeSpecName "kube-api-access-mfhqg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:27:57.025688 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:27:57.025666    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41c42cfc-2480-4bb7-9c60-7da3336dab9f-dshm" (OuterVolumeSpecName: "dshm") pod "41c42cfc-2480-4bb7-9c60-7da3336dab9f" (UID: "41c42cfc-2480-4bb7-9c60-7da3336dab9f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:27:57.046338 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:27:57.046305    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41c42cfc-2480-4bb7-9c60-7da3336dab9f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "41c42cfc-2480-4bb7-9c60-7da3336dab9f" (UID: "41c42cfc-2480-4bb7-9c60-7da3336dab9f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:27:57.124043 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:27:57.124016    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/41c42cfc-2480-4bb7-9c60-7da3336dab9f-kserve-provision-location\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:27:57.124357 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:27:57.124051    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/41c42cfc-2480-4bb7-9c60-7da3336dab9f-model-cache\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:27:57.124357 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:27:57.124065    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/41c42cfc-2480-4bb7-9c60-7da3336dab9f-dshm\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:27:57.124357 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:27:57.124080    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mfhqg\" (UniqueName: \"kubernetes.io/projected/41c42cfc-2480-4bb7-9c60-7da3336dab9f-kube-api-access-mfhqg\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:27:57.124357 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:27:57.124093    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41c42cfc-2480-4bb7-9c60-7da3336dab9f-tls-certs\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:27:57.124357 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:27:57.124101    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/41c42cfc-2480-4bb7-9c60-7da3336dab9f-cabundle-cert\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:27:57.128882 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:27:57.128857    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-6b68df8465-mv8ft"
Aug 04 18:27:57.224954 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:27:57.224923    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/083f6773-b9f4-4bcf-894a-9eb325e0c4d9-kserve-provision-location\") pod \"083f6773-b9f4-4bcf-894a-9eb325e0c4d9\" (UID: \"083f6773-b9f4-4bcf-894a-9eb325e0c4d9\") "
Aug 04 18:27:57.224954 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:27:57.224959    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/083f6773-b9f4-4bcf-894a-9eb325e0c4d9-tmp-dir\") pod \"083f6773-b9f4-4bcf-894a-9eb325e0c4d9\" (UID: \"083f6773-b9f4-4bcf-894a-9eb325e0c4d9\") "
Aug 04 18:27:57.225194 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:27:57.224975    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/083f6773-b9f4-4bcf-894a-9eb325e0c4d9-tls-certs\") pod \"083f6773-b9f4-4bcf-894a-9eb325e0c4d9\" (UID: \"083f6773-b9f4-4bcf-894a-9eb325e0c4d9\") "
Aug 04 18:27:57.225194 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:27:57.225016    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/083f6773-b9f4-4bcf-894a-9eb325e0c4d9-cabundle-cert\") pod \"083f6773-b9f4-4bcf-894a-9eb325e0c4d9\" (UID: \"083f6773-b9f4-4bcf-894a-9eb325e0c4d9\") "
Aug 04 18:27:57.225334 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:27:57.225301    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/083f6773-b9f4-4bcf-894a-9eb325e0c4d9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "083f6773-b9f4-4bcf-894a-9eb325e0c4d9" (UID: "083f6773-b9f4-4bcf-894a-9eb325e0c4d9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:27:57.225447 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:27:57.225425    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/083f6773-b9f4-4bcf-894a-9eb325e0c4d9-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "083f6773-b9f4-4bcf-894a-9eb325e0c4d9" (UID: "083f6773-b9f4-4bcf-894a-9eb325e0c4d9"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:27:57.227018 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:27:57.226997    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/083f6773-b9f4-4bcf-894a-9eb325e0c4d9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "083f6773-b9f4-4bcf-894a-9eb325e0c4d9" (UID: "083f6773-b9f4-4bcf-894a-9eb325e0c4d9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:27:57.242455 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:27:57.242426    2583 generic.go:356] "Generic (PLEG): container finished" podID="083f6773-b9f4-4bcf-894a-9eb325e0c4d9" containerID="327da06461652632319b64ad73178fac702dbd31746828cd63ee9df185268e84" exitCode=0
Aug 04 18:27:57.242611 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:27:57.242496    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-6b68df8465-mv8ft"
Aug 04 18:27:57.242611 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:27:57.242516    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-6b68df8465-mv8ft" event={"ID":"083f6773-b9f4-4bcf-894a-9eb325e0c4d9","Type":"ContainerDied","Data":"327da06461652632319b64ad73178fac702dbd31746828cd63ee9df185268e84"}
Aug 04 18:27:57.242611 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:27:57.242585    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-6b68df8465-mv8ft" event={"ID":"083f6773-b9f4-4bcf-894a-9eb325e0c4d9","Type":"ContainerDied","Data":"f2f7e8f2f44180d0d4b14d64eea19050f8e7844c8207c9ab750e5cc9fa4fbb91"}
Aug 04 18:27:57.242611 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:27:57.242609    2583 scope.go:117] "RemoveContainer" containerID="327da06461652632319b64ad73178fac702dbd31746828cd63ee9df185268e84"
Aug 04 18:27:57.244314 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:27:57.244291    2583 generic.go:356] "Generic (PLEG): container finished" podID="41c42cfc-2480-4bb7-9c60-7da3336dab9f" containerID="77a6c198f9322eb992eff4f910e15baeab7f437291d3fef21af3c1ac6357275b" exitCode=0
Aug 04 18:27:57.244419 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:27:57.244318    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9d7889df5-6c2xx" event={"ID":"41c42cfc-2480-4bb7-9c60-7da3336dab9f","Type":"ContainerDied","Data":"77a6c198f9322eb992eff4f910e15baeab7f437291d3fef21af3c1ac6357275b"}
Aug 04 18:27:57.244419 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:27:57.244366    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9d7889df5-6c2xx" event={"ID":"41c42cfc-2480-4bb7-9c60-7da3336dab9f","Type":"ContainerDied","Data":"a7a7fb3498999cd3ab1d7ffcc67a5a1553a26a308ef84b51a4bb52b7fcbf896e"}
Aug 04 18:27:57.244419 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:27:57.244371    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9d7889df5-6c2xx"
Aug 04 18:27:57.251050 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:27:57.251023    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/083f6773-b9f4-4bcf-894a-9eb325e0c4d9-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "083f6773-b9f4-4bcf-894a-9eb325e0c4d9" (UID: "083f6773-b9f4-4bcf-894a-9eb325e0c4d9"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:27:57.252523 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:27:57.252501    2583 scope.go:117] "RemoveContainer" containerID="aeabdf54cdc44e12dc3ddebd5b34fff5e77ff44ab34665c185ccb9dc20f897a3"
Aug 04 18:27:57.260934 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:27:57.260919    2583 scope.go:117] "RemoveContainer" containerID="327da06461652632319b64ad73178fac702dbd31746828cd63ee9df185268e84"
Aug 04 18:27:57.261155 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:27:57.261137    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"327da06461652632319b64ad73178fac702dbd31746828cd63ee9df185268e84\": container with ID starting with 327da06461652632319b64ad73178fac702dbd31746828cd63ee9df185268e84 not found: ID does not exist" containerID="327da06461652632319b64ad73178fac702dbd31746828cd63ee9df185268e84"
Aug 04 18:27:57.261201 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:27:57.261162    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"327da06461652632319b64ad73178fac702dbd31746828cd63ee9df185268e84"} err="failed to get container status \"327da06461652632319b64ad73178fac702dbd31746828cd63ee9df185268e84\": rpc error: code = NotFound desc = could not find container \"327da06461652632319b64ad73178fac702dbd31746828cd63ee9df185268e84\": container with ID starting with 327da06461652632319b64ad73178fac702dbd31746828cd63ee9df185268e84 not found: ID does not exist"
Aug 04 18:27:57.261201 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:27:57.261178    2583 scope.go:117] "RemoveContainer" containerID="aeabdf54cdc44e12dc3ddebd5b34fff5e77ff44ab34665c185ccb9dc20f897a3"
Aug 04 18:27:57.261373 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:27:57.261355    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aeabdf54cdc44e12dc3ddebd5b34fff5e77ff44ab34665c185ccb9dc20f897a3\": container with ID starting with aeabdf54cdc44e12dc3ddebd5b34fff5e77ff44ab34665c185ccb9dc20f897a3 not found: ID does not exist" containerID="aeabdf54cdc44e12dc3ddebd5b34fff5e77ff44ab34665c185ccb9dc20f897a3"
Aug 04 18:27:57.261420 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:27:57.261378    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aeabdf54cdc44e12dc3ddebd5b34fff5e77ff44ab34665c185ccb9dc20f897a3"} err="failed to get container status \"aeabdf54cdc44e12dc3ddebd5b34fff5e77ff44ab34665c185ccb9dc20f897a3\": rpc error: code = NotFound desc = could not find container \"aeabdf54cdc44e12dc3ddebd5b34fff5e77ff44ab34665c185ccb9dc20f897a3\": container with ID starting with aeabdf54cdc44e12dc3ddebd5b34fff5e77ff44ab34665c185ccb9dc20f897a3 not found: ID does not exist"
Aug 04 18:27:57.261420 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:27:57.261396    2583 scope.go:117] "RemoveContainer" containerID="77a6c198f9322eb992eff4f910e15baeab7f437291d3fef21af3c1ac6357275b"
Aug 04 18:27:57.268649 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:27:57.268626    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9d7889df5-6c2xx"]
Aug 04 18:27:57.269632 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:27:57.269613    2583 scope.go:117] "RemoveContainer" containerID="73b193b7971488a45dd2c63bd4363a921b78549f222dc5f53d49851ae8fd41be"
Aug 04 18:27:57.275198 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:27:57.275177    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-9d7889df5-6c2xx"]
Aug 04 18:27:57.277529 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:27:57.277513    2583 scope.go:117] "RemoveContainer" containerID="77a6c198f9322eb992eff4f910e15baeab7f437291d3fef21af3c1ac6357275b"
Aug 04 18:27:57.277764 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:27:57.277747    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77a6c198f9322eb992eff4f910e15baeab7f437291d3fef21af3c1ac6357275b\": container with ID starting with 77a6c198f9322eb992eff4f910e15baeab7f437291d3fef21af3c1ac6357275b not found: ID does not exist" containerID="77a6c198f9322eb992eff4f910e15baeab7f437291d3fef21af3c1ac6357275b"
Aug 04 18:27:57.277856 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:27:57.277773    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77a6c198f9322eb992eff4f910e15baeab7f437291d3fef21af3c1ac6357275b"} err="failed to get container status \"77a6c198f9322eb992eff4f910e15baeab7f437291d3fef21af3c1ac6357275b\": rpc error: code = NotFound desc = could not find container \"77a6c198f9322eb992eff4f910e15baeab7f437291d3fef21af3c1ac6357275b\": container with ID starting with 77a6c198f9322eb992eff4f910e15baeab7f437291d3fef21af3c1ac6357275b not found: ID does not exist"
Aug 04 18:27:57.277856 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:27:57.277789    2583 scope.go:117] "RemoveContainer" containerID="73b193b7971488a45dd2c63bd4363a921b78549f222dc5f53d49851ae8fd41be"
Aug 04 18:27:57.278028 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:27:57.278011    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73b193b7971488a45dd2c63bd4363a921b78549f222dc5f53d49851ae8fd41be\": container with ID starting with 73b193b7971488a45dd2c63bd4363a921b78549f222dc5f53d49851ae8fd41be not found: ID does not exist" containerID="73b193b7971488a45dd2c63bd4363a921b78549f222dc5f53d49851ae8fd41be"
Aug 04 18:27:57.278068 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:27:57.278034    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73b193b7971488a45dd2c63bd4363a921b78549f222dc5f53d49851ae8fd41be"} err="failed to get container status \"73b193b7971488a45dd2c63bd4363a921b78549f222dc5f53d49851ae8fd41be\": rpc error: code = NotFound desc = could not find container \"73b193b7971488a45dd2c63bd4363a921b78549f222dc5f53d49851ae8fd41be\": container with ID starting with 73b193b7971488a45dd2c63bd4363a921b78549f222dc5f53d49851ae8fd41be not found: ID does not exist"
Aug 04 18:27:57.325608 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:27:57.325589    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/083f6773-b9f4-4bcf-894a-9eb325e0c4d9-kserve-provision-location\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:27:57.325608 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:27:57.325607    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/083f6773-b9f4-4bcf-894a-9eb325e0c4d9-tmp-dir\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:27:57.325744 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:27:57.325618    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/083f6773-b9f4-4bcf-894a-9eb325e0c4d9-tls-certs\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:27:57.325744 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:27:57.325626    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/083f6773-b9f4-4bcf-894a-9eb325e0c4d9-cabundle-cert\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:27:57.566869 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:27:57.566836    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-6b68df8465-mv8ft"]
Aug 04 18:27:57.572600 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:27:57.572576    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-6b68df8465-mv8ft"]
Aug 04 18:27:58.944033 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:27:58.944000    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="083f6773-b9f4-4bcf-894a-9eb325e0c4d9" path="/var/lib/kubelet/pods/083f6773-b9f4-4bcf-894a-9eb325e0c4d9/volumes"
Aug 04 18:27:58.944481 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:27:58.944456    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="41c42cfc-2480-4bb7-9c60-7da3336dab9f" path="/var/lib/kubelet/pods/41c42cfc-2480-4bb7-9c60-7da3336dab9f/volumes"
Aug 04 18:28:13.103653 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:28:13.103571    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-2hzr2"]
Aug 04 18:28:13.104221 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:28:13.103916    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41c42cfc-2480-4bb7-9c60-7da3336dab9f" containerName="storage-initializer"
Aug 04 18:28:13.104221 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:28:13.103927    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="41c42cfc-2480-4bb7-9c60-7da3336dab9f" containerName="storage-initializer"
Aug 04 18:28:13.104221 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:28:13.103940    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="083f6773-b9f4-4bcf-894a-9eb325e0c4d9" containerName="main"
Aug 04 18:28:13.104221 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:28:13.103946    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="083f6773-b9f4-4bcf-894a-9eb325e0c4d9" containerName="main"
Aug 04 18:28:13.104221 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:28:13.103953    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="083f6773-b9f4-4bcf-894a-9eb325e0c4d9" containerName="storage-initializer"
Aug 04 18:28:13.104221 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:28:13.103959    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="083f6773-b9f4-4bcf-894a-9eb325e0c4d9" containerName="storage-initializer"
Aug 04 18:28:13.104221 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:28:13.103967    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41c42cfc-2480-4bb7-9c60-7da3336dab9f" containerName="main"
Aug 04 18:28:13.104221 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:28:13.103973    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="41c42cfc-2480-4bb7-9c60-7da3336dab9f" containerName="main"
Aug 04 18:28:13.104221 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:28:13.103979    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0bd3381e-6544-4ad5-86ad-2bdf37e2558c" containerName="main"
Aug 04 18:28:13.104221 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:28:13.103984    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="0bd3381e-6544-4ad5-86ad-2bdf37e2558c" containerName="main"
Aug 04 18:28:13.104221 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:28:13.104034    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="0bd3381e-6544-4ad5-86ad-2bdf37e2558c" containerName="main"
Aug 04 18:28:13.104221 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:28:13.104041    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="41c42cfc-2480-4bb7-9c60-7da3336dab9f" containerName="main"
Aug 04 18:28:13.104221 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:28:13.104048    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="083f6773-b9f4-4bcf-894a-9eb325e0c4d9" containerName="main"
Aug 04 18:28:13.108749 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:28:13.108726    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-2hzr2"
Aug 04 18:28:13.111722 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:28:13.111702    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-model-name-divergence-resolves-b3ec91d2\"/\"openshift-service-ca.crt\""
Aug 04 18:28:13.113003 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:28:13.112982    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-model-name-divergence-resolves-b3ec91d2\"/\"default-dockercfg-cxsbt\""
Aug 04 18:28:13.113238 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:28:13.113220    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-model-name-divergence-resolves-b3ec91d2\"/\"kube-root-ca.crt\""
Aug 04 18:28:13.113362 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:28:13.113204    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-model-name-divergence-resolves-b3ec91d2\"/\"test-div-v1-kserve-self-signed-certs\""
Aug 04 18:28:13.116032 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:28:13.116008    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-2hzr2"]
Aug 04 18:28:13.157672 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:28:13.157635    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4dd6117-515a-4eab-9975-83bb59758255-model-cache\") pod \"test-div-v1-kserve-fb566d667-2hzr2\" (UID: \"b4dd6117-515a-4eab-9975-83bb59758255\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-2hzr2"
Aug 04 18:28:13.157672 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:28:13.157668    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4dd6117-515a-4eab-9975-83bb59758255-tmp-dir\") pod \"test-div-v1-kserve-fb566d667-2hzr2\" (UID: \"b4dd6117-515a-4eab-9975-83bb59758255\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-2hzr2"
Aug 04 18:28:13.157903 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:28:13.157769    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b4dd6117-515a-4eab-9975-83bb59758255-home\") pod \"test-div-v1-kserve-fb566d667-2hzr2\" (UID: \"b4dd6117-515a-4eab-9975-83bb59758255\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-2hzr2"
Aug 04 18:28:13.157903 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:28:13.157812    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbxqx\" (UniqueName: \"kubernetes.io/projected/b4dd6117-515a-4eab-9975-83bb59758255-kube-api-access-hbxqx\") pod \"test-div-v1-kserve-fb566d667-2hzr2\" (UID: \"b4dd6117-515a-4eab-9975-83bb59758255\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-2hzr2"
Aug 04 18:28:13.157903 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:28:13.157859    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4dd6117-515a-4eab-9975-83bb59758255-tls-certs\") pod \"test-div-v1-kserve-fb566d667-2hzr2\" (UID: \"b4dd6117-515a-4eab-9975-83bb59758255\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-2hzr2"
Aug 04 18:28:13.157903 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:28:13.157901    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4dd6117-515a-4eab-9975-83bb59758255-dshm\") pod \"test-div-v1-kserve-fb566d667-2hzr2\" (UID: \"b4dd6117-515a-4eab-9975-83bb59758255\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-2hzr2"
Aug 04 18:28:13.259244 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:28:13.259207    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b4dd6117-515a-4eab-9975-83bb59758255-home\") pod \"test-div-v1-kserve-fb566d667-2hzr2\" (UID: \"b4dd6117-515a-4eab-9975-83bb59758255\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-2hzr2"
Aug 04 18:28:13.259244 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:28:13.259248    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbxqx\" (UniqueName: \"kubernetes.io/projected/b4dd6117-515a-4eab-9975-83bb59758255-kube-api-access-hbxqx\") pod \"test-div-v1-kserve-fb566d667-2hzr2\" (UID: \"b4dd6117-515a-4eab-9975-83bb59758255\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-2hzr2"
Aug 04 18:28:13.259563 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:28:13.259286    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4dd6117-515a-4eab-9975-83bb59758255-tls-certs\") pod \"test-div-v1-kserve-fb566d667-2hzr2\" (UID: \"b4dd6117-515a-4eab-9975-83bb59758255\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-2hzr2"
Aug 04 18:28:13.259563 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:28:13.259326    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4dd6117-515a-4eab-9975-83bb59758255-dshm\") pod \"test-div-v1-kserve-fb566d667-2hzr2\" (UID: \"b4dd6117-515a-4eab-9975-83bb59758255\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-2hzr2"
Aug 04 18:28:13.259563 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:28:13.259359    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4dd6117-515a-4eab-9975-83bb59758255-model-cache\") pod \"test-div-v1-kserve-fb566d667-2hzr2\" (UID: \"b4dd6117-515a-4eab-9975-83bb59758255\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-2hzr2"
Aug 04 18:28:13.259563 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:28:13.259381    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4dd6117-515a-4eab-9975-83bb59758255-tmp-dir\") pod \"test-div-v1-kserve-fb566d667-2hzr2\" (UID: \"b4dd6117-515a-4eab-9975-83bb59758255\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-2hzr2"
Aug 04 18:28:13.259791 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:28:13.259761    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b4dd6117-515a-4eab-9975-83bb59758255-home\") pod \"test-div-v1-kserve-fb566d667-2hzr2\" (UID: \"b4dd6117-515a-4eab-9975-83bb59758255\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-2hzr2"
Aug 04 18:28:13.259955 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:28:13.259925    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4dd6117-515a-4eab-9975-83bb59758255-tmp-dir\") pod \"test-div-v1-kserve-fb566d667-2hzr2\" (UID: \"b4dd6117-515a-4eab-9975-83bb59758255\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-2hzr2"
Aug 04 18:28:13.260057 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:28:13.260041    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4dd6117-515a-4eab-9975-83bb59758255-model-cache\") pod \"test-div-v1-kserve-fb566d667-2hzr2\" (UID: \"b4dd6117-515a-4eab-9975-83bb59758255\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-2hzr2"
Aug 04 18:28:13.261765 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:28:13.261747    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4dd6117-515a-4eab-9975-83bb59758255-dshm\") pod \"test-div-v1-kserve-fb566d667-2hzr2\" (UID: \"b4dd6117-515a-4eab-9975-83bb59758255\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-2hzr2"
Aug 04 18:28:13.262061 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:28:13.262038    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4dd6117-515a-4eab-9975-83bb59758255-tls-certs\") pod \"test-div-v1-kserve-fb566d667-2hzr2\" (UID: \"b4dd6117-515a-4eab-9975-83bb59758255\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-2hzr2"
Aug 04 18:28:13.267510 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:28:13.267491    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbxqx\" (UniqueName: \"kubernetes.io/projected/b4dd6117-515a-4eab-9975-83bb59758255-kube-api-access-hbxqx\") pod \"test-div-v1-kserve-fb566d667-2hzr2\" (UID: \"b4dd6117-515a-4eab-9975-83bb59758255\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-2hzr2"
Aug 04 18:28:13.421658 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:28:13.421574    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-2hzr2"
Aug 04 18:28:13.564396 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:28:13.564371    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-2hzr2"]
Aug 04 18:28:13.569959 ip-10-0-135-124 kubenswrapper[2583]: W0804 18:28:13.567875    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4dd6117_515a_4eab_9975_83bb59758255.slice/crio-0c2d3054bbeb8b68972f817a8c397513c4b9f950a7c23e15dfebc0bcf9747e95 WatchSource:0}: Error finding container 0c2d3054bbeb8b68972f817a8c397513c4b9f950a7c23e15dfebc0bcf9747e95: Status 404 returned error can't find the container with id 0c2d3054bbeb8b68972f817a8c397513c4b9f950a7c23e15dfebc0bcf9747e95
Aug 04 18:28:14.337799 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:28:14.337758    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-2hzr2" event={"ID":"b4dd6117-515a-4eab-9975-83bb59758255","Type":"ContainerStarted","Data":"c6d3e5715ce4d4fce9366f215cd397dcf6ee10caa7dcc668c98d8533356be279"}
Aug 04 18:28:14.338281 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:28:14.337856    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-2hzr2" event={"ID":"b4dd6117-515a-4eab-9975-83bb59758255","Type":"ContainerStarted","Data":"0c2d3054bbeb8b68972f817a8c397513c4b9f950a7c23e15dfebc0bcf9747e95"}
Aug 04 18:28:14.358031 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:28:14.357985    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-2hzr2" podStartSLOduration=1.3579726349999999 podStartE2EDuration="1.357972635s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:28:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:28:14.357196418 +0000 UTC m=+1707.938414995" watchObservedRunningTime="2026-08-04 18:28:14.357972635 +0000 UTC m=+1707.939191212"
Aug 04 18:28:14.421991 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:28:14.421962    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-2hzr2"
Aug 04 18:28:23.422361 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:28:23.422319    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-2hzr2"
Aug 04 18:28:23.435651 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:28:23.435614    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-2hzr2"
Aug 04 18:28:24.400566 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:28:24.400534    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-2hzr2"
Aug 04 18:28:42.036058 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:28:42.036027    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6d5c87f9b9-5zl75"]
Aug 04 18:28:42.039725 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:28:42.039703    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6d5c87f9b9-5zl75"
Aug 04 18:28:42.042569 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:28:42.042542    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"odh-kserve-custom-ca-bundle\""
Aug 04 18:28:42.042688 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:28:42.042572    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"seaweedfs-s3-creds\""
Aug 04 18:28:42.042688 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:28:42.042657    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"default-dockercfg-9qrb5\""
Aug 04 18:28:42.043630 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:28:42.043609    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"tokenizer-template-args-test-kserve-self-signed-certs\""
Aug 04 18:28:42.043717 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:28:42.043673    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"kube-root-ca.crt\""
Aug 04 18:28:42.043717 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:28:42.043687    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"openshift-service-ca.crt\""
Aug 04 18:28:42.052867 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:28:42.052846    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6d5c87f9b9-5zl75"]
Aug 04 18:28:42.194399 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:28:42.194368    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/beb61a8e-02b7-48a8-bdef-5675f4095424-model-cache\") pod \"tokenizer-template-args-test-kserve-6d5c87f9b9-5zl75\" (UID: \"beb61a8e-02b7-48a8-bdef-5675f4095424\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6d5c87f9b9-5zl75"
Aug 04 18:28:42.194580 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:28:42.194405    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/beb61a8e-02b7-48a8-bdef-5675f4095424-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-6d5c87f9b9-5zl75\" (UID: \"beb61a8e-02b7-48a8-bdef-5675f4095424\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6d5c87f9b9-5zl75"
Aug 04 18:28:42.194580 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:28:42.194435    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/beb61a8e-02b7-48a8-bdef-5675f4095424-tls-certs\") pod \"tokenizer-template-args-test-kserve-6d5c87f9b9-5zl75\" (UID: \"beb61a8e-02b7-48a8-bdef-5675f4095424\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6d5c87f9b9-5zl75"
Aug 04 18:28:42.194580 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:28:42.194539    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/beb61a8e-02b7-48a8-bdef-5675f4095424-tmp-dir\") pod \"tokenizer-template-args-test-kserve-6d5c87f9b9-5zl75\" (UID: \"beb61a8e-02b7-48a8-bdef-5675f4095424\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6d5c87f9b9-5zl75"
Aug 04 18:28:42.194580 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:28:42.194573    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/beb61a8e-02b7-48a8-bdef-5675f4095424-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-6d5c87f9b9-5zl75\" (UID: \"beb61a8e-02b7-48a8-bdef-5675f4095424\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6d5c87f9b9-5zl75"
Aug 04 18:28:42.194721 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:28:42.194597    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l79n4\" (UniqueName: \"kubernetes.io/projected/beb61a8e-02b7-48a8-bdef-5675f4095424-kube-api-access-l79n4\") pod \"tokenizer-template-args-test-kserve-6d5c87f9b9-5zl75\" (UID: \"beb61a8e-02b7-48a8-bdef-5675f4095424\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6d5c87f9b9-5zl75"
Aug 04 18:28:42.194721 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:28:42.194633    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/beb61a8e-02b7-48a8-bdef-5675f4095424-dshm\") pod \"tokenizer-template-args-test-kserve-6d5c87f9b9-5zl75\" (UID: \"beb61a8e-02b7-48a8-bdef-5675f4095424\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6d5c87f9b9-5zl75"
Aug 04 18:28:42.194721 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:28:42.194651    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/beb61a8e-02b7-48a8-bdef-5675f4095424-home\") pod \"tokenizer-template-args-test-kserve-6d5c87f9b9-5zl75\" (UID: \"beb61a8e-02b7-48a8-bdef-5675f4095424\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6d5c87f9b9-5zl75"
Aug 04 18:28:42.295269 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:28:42.295176    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/beb61a8e-02b7-48a8-bdef-5675f4095424-model-cache\") pod \"tokenizer-template-args-test-kserve-6d5c87f9b9-5zl75\" (UID: \"beb61a8e-02b7-48a8-bdef-5675f4095424\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6d5c87f9b9-5zl75"
Aug 04 18:28:42.295269 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:28:42.295252    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/beb61a8e-02b7-48a8-bdef-5675f4095424-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-6d5c87f9b9-5zl75\" (UID: \"beb61a8e-02b7-48a8-bdef-5675f4095424\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6d5c87f9b9-5zl75"
Aug 04 18:28:42.295512 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:28:42.295285    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/beb61a8e-02b7-48a8-bdef-5675f4095424-tls-certs\") pod \"tokenizer-template-args-test-kserve-6d5c87f9b9-5zl75\" (UID: \"beb61a8e-02b7-48a8-bdef-5675f4095424\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6d5c87f9b9-5zl75"
Aug 04 18:28:42.295512 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:28:42.295332    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/beb61a8e-02b7-48a8-bdef-5675f4095424-tmp-dir\") pod \"tokenizer-template-args-test-kserve-6d5c87f9b9-5zl75\" (UID: \"beb61a8e-02b7-48a8-bdef-5675f4095424\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6d5c87f9b9-5zl75"
Aug 04 18:28:42.295512 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:28:42.295353    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/beb61a8e-02b7-48a8-bdef-5675f4095424-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-6d5c87f9b9-5zl75\" (UID: \"beb61a8e-02b7-48a8-bdef-5675f4095424\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6d5c87f9b9-5zl75"
Aug 04 18:28:42.295512 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:28:42.295380    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l79n4\" (UniqueName: \"kubernetes.io/projected/beb61a8e-02b7-48a8-bdef-5675f4095424-kube-api-access-l79n4\") pod \"tokenizer-template-args-test-kserve-6d5c87f9b9-5zl75\" (UID: \"beb61a8e-02b7-48a8-bdef-5675f4095424\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6d5c87f9b9-5zl75"
Aug 04 18:28:42.295512 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:28:42.295402    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/beb61a8e-02b7-48a8-bdef-5675f4095424-dshm\") pod \"tokenizer-template-args-test-kserve-6d5c87f9b9-5zl75\" (UID: \"beb61a8e-02b7-48a8-bdef-5675f4095424\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6d5c87f9b9-5zl75"
Aug 04 18:28:42.295512 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:28:42.295420    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/beb61a8e-02b7-48a8-bdef-5675f4095424-home\") pod \"tokenizer-template-args-test-kserve-6d5c87f9b9-5zl75\" (UID: \"beb61a8e-02b7-48a8-bdef-5675f4095424\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6d5c87f9b9-5zl75"
Aug 04 18:28:42.295852 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:28:42.295620    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/beb61a8e-02b7-48a8-bdef-5675f4095424-model-cache\") pod \"tokenizer-template-args-test-kserve-6d5c87f9b9-5zl75\" (UID: \"beb61a8e-02b7-48a8-bdef-5675f4095424\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6d5c87f9b9-5zl75"
Aug 04 18:28:42.295913 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:28:42.295888    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/beb61a8e-02b7-48a8-bdef-5675f4095424-tmp-dir\") pod \"tokenizer-template-args-test-kserve-6d5c87f9b9-5zl75\" (UID: \"beb61a8e-02b7-48a8-bdef-5675f4095424\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6d5c87f9b9-5zl75"
Aug 04 18:28:42.296044 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:28:42.296022    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/beb61a8e-02b7-48a8-bdef-5675f4095424-home\") pod \"tokenizer-template-args-test-kserve-6d5c87f9b9-5zl75\" (UID: \"beb61a8e-02b7-48a8-bdef-5675f4095424\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6d5c87f9b9-5zl75"
Aug 04 18:28:42.296443 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:28:42.296411    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/beb61a8e-02b7-48a8-bdef-5675f4095424-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-6d5c87f9b9-5zl75\" (UID: \"beb61a8e-02b7-48a8-bdef-5675f4095424\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6d5c87f9b9-5zl75"
Aug 04 18:28:42.296668 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:28:42.296637    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/beb61a8e-02b7-48a8-bdef-5675f4095424-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-6d5c87f9b9-5zl75\" (UID: \"beb61a8e-02b7-48a8-bdef-5675f4095424\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6d5c87f9b9-5zl75"
Aug 04 18:28:42.298176 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:28:42.298153    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/beb61a8e-02b7-48a8-bdef-5675f4095424-dshm\") pod \"tokenizer-template-args-test-kserve-6d5c87f9b9-5zl75\" (UID: \"beb61a8e-02b7-48a8-bdef-5675f4095424\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6d5c87f9b9-5zl75"
Aug 04 18:28:42.298279 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:28:42.298207    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/beb61a8e-02b7-48a8-bdef-5675f4095424-tls-certs\") pod \"tokenizer-template-args-test-kserve-6d5c87f9b9-5zl75\" (UID: \"beb61a8e-02b7-48a8-bdef-5675f4095424\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6d5c87f9b9-5zl75"
Aug 04 18:28:42.299391 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:28:42.299367    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-56469b8ffc-nr4f4"]
Aug 04 18:28:42.306515 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:28:42.305194    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-56469b8ffc-nr4f4"
Aug 04 18:28:42.313856 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:28:42.313829    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l79n4\" (UniqueName: \"kubernetes.io/projected/beb61a8e-02b7-48a8-bdef-5675f4095424-kube-api-access-l79n4\") pod \"tokenizer-template-args-test-kserve-6d5c87f9b9-5zl75\" (UID: \"beb61a8e-02b7-48a8-bdef-5675f4095424\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6d5c87f9b9-5zl75"
Aug 04 18:28:42.318397 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:28:42.318372    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-56469b8ffc-nr4f4"]
Aug 04 18:28:42.350608 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:28:42.350580    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6d5c87f9b9-5zl75"
Aug 04 18:28:42.396894 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:28:42.396859    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/813549a1-df99-46b7-a304-8b57329fdee8-tmp-dir\") pod \"tokenizer-template-args-test-tokenizer-56469b8ffc-nr4f4\" (UID: \"813549a1-df99-46b7-a304-8b57329fdee8\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-56469b8ffc-nr4f4"
Aug 04 18:28:42.397059 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:28:42.396920    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/813549a1-df99-46b7-a304-8b57329fdee8-kserve-provision-location\") pod \"tokenizer-template-args-test-tokenizer-56469b8ffc-nr4f4\" (UID: \"813549a1-df99-46b7-a304-8b57329fdee8\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-56469b8ffc-nr4f4"
Aug 04 18:28:42.397059 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:28:42.397014    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/813549a1-df99-46b7-a304-8b57329fdee8-cabundle-cert\") pod \"tokenizer-template-args-test-tokenizer-56469b8ffc-nr4f4\" (UID: \"813549a1-df99-46b7-a304-8b57329fdee8\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-56469b8ffc-nr4f4"
Aug 04 18:28:42.397170 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:28:42.397073    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/813549a1-df99-46b7-a304-8b57329fdee8-tls-certs\") pod \"tokenizer-template-args-test-tokenizer-56469b8ffc-nr4f4\" (UID: \"813549a1-df99-46b7-a304-8b57329fdee8\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-56469b8ffc-nr4f4"
Aug 04 18:28:42.488545 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:28:42.488511    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6d5c87f9b9-5zl75"]
Aug 04 18:28:42.489982 ip-10-0-135-124 kubenswrapper[2583]: W0804 18:28:42.489956    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbeb61a8e_02b7_48a8_bdef_5675f4095424.slice/crio-b562fb037ba21472dd9f8bf4f9a9e9f6acfbf58653aaabc3adb34bdedc032abd WatchSource:0}: Error finding container b562fb037ba21472dd9f8bf4f9a9e9f6acfbf58653aaabc3adb34bdedc032abd: Status 404 returned error can't find the container with id b562fb037ba21472dd9f8bf4f9a9e9f6acfbf58653aaabc3adb34bdedc032abd
Aug 04 18:28:42.497802 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:28:42.497777    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/813549a1-df99-46b7-a304-8b57329fdee8-cabundle-cert\") pod \"tokenizer-template-args-test-tokenizer-56469b8ffc-nr4f4\" (UID: \"813549a1-df99-46b7-a304-8b57329fdee8\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-56469b8ffc-nr4f4"
Aug 04 18:28:42.497896 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:28:42.497837    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/813549a1-df99-46b7-a304-8b57329fdee8-tls-certs\") pod \"tokenizer-template-args-test-tokenizer-56469b8ffc-nr4f4\" (UID: \"813549a1-df99-46b7-a304-8b57329fdee8\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-56469b8ffc-nr4f4"
Aug 04 18:28:42.497896 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:28:42.497873    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/813549a1-df99-46b7-a304-8b57329fdee8-tmp-dir\") pod \"tokenizer-template-args-test-tokenizer-56469b8ffc-nr4f4\" (UID: \"813549a1-df99-46b7-a304-8b57329fdee8\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-56469b8ffc-nr4f4"
Aug 04 18:28:42.498011 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:28:42.497903    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/813549a1-df99-46b7-a304-8b57329fdee8-kserve-provision-location\") pod \"tokenizer-template-args-test-tokenizer-56469b8ffc-nr4f4\" (UID: \"813549a1-df99-46b7-a304-8b57329fdee8\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-56469b8ffc-nr4f4"
Aug 04 18:28:42.498246 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:28:42.498224    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/813549a1-df99-46b7-a304-8b57329fdee8-tmp-dir\") pod \"tokenizer-template-args-test-tokenizer-56469b8ffc-nr4f4\" (UID: \"813549a1-df99-46b7-a304-8b57329fdee8\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-56469b8ffc-nr4f4"
Aug 04 18:28:42.498295 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:28:42.498276    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/813549a1-df99-46b7-a304-8b57329fdee8-kserve-provision-location\") pod \"tokenizer-template-args-test-tokenizer-56469b8ffc-nr4f4\" (UID: \"813549a1-df99-46b7-a304-8b57329fdee8\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-56469b8ffc-nr4f4"
Aug 04 18:28:42.498528 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:28:42.498511    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/813549a1-df99-46b7-a304-8b57329fdee8-cabundle-cert\") pod \"tokenizer-template-args-test-tokenizer-56469b8ffc-nr4f4\" (UID: \"813549a1-df99-46b7-a304-8b57329fdee8\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-56469b8ffc-nr4f4"
Aug 04 18:28:42.500345 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:28:42.500327    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/813549a1-df99-46b7-a304-8b57329fdee8-tls-certs\") pod \"tokenizer-template-args-test-tokenizer-56469b8ffc-nr4f4\" (UID: \"813549a1-df99-46b7-a304-8b57329fdee8\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-56469b8ffc-nr4f4"
Aug 04 18:28:42.637210 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:28:42.637130    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-56469b8ffc-nr4f4"
Aug 04 18:28:42.778395 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:28:42.778364    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-56469b8ffc-nr4f4"]
Aug 04 18:28:42.780568 ip-10-0-135-124 kubenswrapper[2583]: W0804 18:28:42.780528    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod813549a1_df99_46b7_a304_8b57329fdee8.slice/crio-eab47b9266f6e527b2d0d9490c3c55c61279846c210fcf8f2f3c73cffb8786e8 WatchSource:0}: Error finding container eab47b9266f6e527b2d0d9490c3c55c61279846c210fcf8f2f3c73cffb8786e8: Status 404 returned error can't find the container with id eab47b9266f6e527b2d0d9490c3c55c61279846c210fcf8f2f3c73cffb8786e8
Aug 04 18:28:43.491862 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:28:43.491820    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6d5c87f9b9-5zl75" event={"ID":"beb61a8e-02b7-48a8-bdef-5675f4095424","Type":"ContainerStarted","Data":"9291f6c5a81b8e84ea6ccfe9b04f264c95bf98fdbf2965b35566ab8c3f8f8cf7"}
Aug 04 18:28:43.492336 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:28:43.491889    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6d5c87f9b9-5zl75" event={"ID":"beb61a8e-02b7-48a8-bdef-5675f4095424","Type":"ContainerStarted","Data":"b562fb037ba21472dd9f8bf4f9a9e9f6acfbf58653aaabc3adb34bdedc032abd"}
Aug 04 18:28:43.493583 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:28:43.493553    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-56469b8ffc-nr4f4" event={"ID":"813549a1-df99-46b7-a304-8b57329fdee8","Type":"ContainerStarted","Data":"fd2cff1f6bf8fc2fe8430147b2d9844594d00e3eee38286b991e7e89eaf5299a"}
Aug 04 18:28:43.493720 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:28:43.493604    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-56469b8ffc-nr4f4" event={"ID":"813549a1-df99-46b7-a304-8b57329fdee8","Type":"ContainerStarted","Data":"eab47b9266f6e527b2d0d9490c3c55c61279846c210fcf8f2f3c73cffb8786e8"}
Aug 04 18:28:45.508231 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:28:45.508200    2583 generic.go:356] "Generic (PLEG): container finished" podID="beb61a8e-02b7-48a8-bdef-5675f4095424" containerID="9291f6c5a81b8e84ea6ccfe9b04f264c95bf98fdbf2965b35566ab8c3f8f8cf7" exitCode=0
Aug 04 18:28:45.508741 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:28:45.508282    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6d5c87f9b9-5zl75" event={"ID":"beb61a8e-02b7-48a8-bdef-5675f4095424","Type":"ContainerDied","Data":"9291f6c5a81b8e84ea6ccfe9b04f264c95bf98fdbf2965b35566ab8c3f8f8cf7"}
Aug 04 18:28:45.510042 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:28:45.510023    2583 generic.go:356] "Generic (PLEG): container finished" podID="813549a1-df99-46b7-a304-8b57329fdee8" containerID="fd2cff1f6bf8fc2fe8430147b2d9844594d00e3eee38286b991e7e89eaf5299a" exitCode=0
Aug 04 18:28:45.510138 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:28:45.510076    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-56469b8ffc-nr4f4" event={"ID":"813549a1-df99-46b7-a304-8b57329fdee8","Type":"ContainerDied","Data":"fd2cff1f6bf8fc2fe8430147b2d9844594d00e3eee38286b991e7e89eaf5299a"}
Aug 04 18:28:46.521655 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:28:46.521615    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-56469b8ffc-nr4f4" event={"ID":"813549a1-df99-46b7-a304-8b57329fdee8","Type":"ContainerStarted","Data":"520ac651bce20e9a6a4b87315c8d76af3f900fc7e1242f0e0d0155bd6e46ba3e"}
Aug 04 18:28:46.522146 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:28:46.521834    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-56469b8ffc-nr4f4"
Aug 04 18:28:46.523455 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:28:46.523430    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6d5c87f9b9-5zl75" event={"ID":"beb61a8e-02b7-48a8-bdef-5675f4095424","Type":"ContainerStarted","Data":"8d0be2ac640ff396fe5a115221c2b9ffb7547c54106045eff93a1e9f4b11db28"}
Aug 04 18:28:46.543441 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:28:46.543399    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-56469b8ffc-nr4f4" podStartSLOduration=2.113397358 podStartE2EDuration="4.54338396s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.429986602s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:28:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:28:46.542265055 +0000 UTC m=+1740.123483634" watchObservedRunningTime="2026-08-04 18:28:46.54338396 +0000 UTC m=+1740.124602537"
Aug 04 18:28:46.571744 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:28:46.571672    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6d5c87f9b9-5zl75" podStartSLOduration=2.120246823 podStartE2EDuration="4.571656726s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.451409903s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:28:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:28:46.569593648 +0000 UTC m=+1740.150812227" watchObservedRunningTime="2026-08-04 18:28:46.571656726 +0000 UTC m=+1740.152875303"
Aug 04 18:28:47.351557 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:28:47.351520    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6d5c87f9b9-5zl75"
Aug 04 18:28:52.350713 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:28:52.350675    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6d5c87f9b9-5zl75"
Aug 04 18:28:52.366233 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:28:52.366204    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6d5c87f9b9-5zl75"
Aug 04 18:28:52.579185 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:28:52.579155    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6d5c87f9b9-5zl75"
Aug 04 18:29:17.542575 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:29:17.542543    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-56469b8ffc-nr4f4"
Aug 04 18:29:19.719628 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:29:19.719597    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-0eb24714_tokenizer-template-args-test-tokenizer-56469b8ffc-nr4f4_813549a1-df99-46b7-a304-8b57329fdee8/main/0.log"
Aug 04 18:29:21.341053 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:29:21.341016    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-56469b8ffc-nr4f4"]
Aug 04 18:29:21.341575 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:29:21.341217    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-56469b8ffc-nr4f4" podUID="813549a1-df99-46b7-a304-8b57329fdee8" containerName="main" containerID="cri-o://520ac651bce20e9a6a4b87315c8d76af3f900fc7e1242f0e0d0155bd6e46ba3e" gracePeriod=30
Aug 04 18:29:21.349291 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:29:21.349263    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6d5c87f9b9-5zl75"]
Aug 04 18:29:21.359907 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:29:21.359872    2583 log.go:32] "ExecSync cmd from runtime service failed" err=<
Aug 04 18:29:21.359907 ip-10-0-135-124 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Aug 04 18:29:21.359907 ip-10-0-135-124 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Aug 04 18:29:21.359907 ip-10-0-135-124 kubenswrapper[2583]:  > containerID="8d0be2ac640ff396fe5a115221c2b9ffb7547c54106045eff93a1e9f4b11db28" cmd=["/bin/sleep","15"]
Aug 04 18:29:21.360200 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:29:21.359920    2583 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6d5c87f9b9-5zl75" message="" err=<
Aug 04 18:29:21.360200 ip-10-0-135-124 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Aug 04 18:29:21.360200 ip-10-0-135-124 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Aug 04 18:29:21.360200 ip-10-0-135-124 kubenswrapper[2583]:  >
Aug 04 18:29:21.360200 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:29:21.359933    2583 kuberuntime_container.go:755] "PreStop hook failed" err=<
Aug 04 18:29:21.360200 ip-10-0-135-124 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Aug 04 18:29:21.360200 ip-10-0-135-124 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Aug 04 18:29:21.360200 ip-10-0-135-124 kubenswrapper[2583]:  > pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6d5c87f9b9-5zl75" podUID="beb61a8e-02b7-48a8-bdef-5675f4095424" containerName="main" containerID="cri-o://8d0be2ac640ff396fe5a115221c2b9ffb7547c54106045eff93a1e9f4b11db28"
Aug 04 18:29:21.360200 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:29:21.359974    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6d5c87f9b9-5zl75" podUID="beb61a8e-02b7-48a8-bdef-5675f4095424" containerName="main" containerID="cri-o://8d0be2ac640ff396fe5a115221c2b9ffb7547c54106045eff93a1e9f4b11db28" gracePeriod=60
Aug 04 18:29:21.568500 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:29:21.568442    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6d5c87f9b9-5zl75" podUID="beb61a8e-02b7-48a8-bdef-5675f4095424" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": dial tcp 10.132.0.64:8000: connect: connection refused"
Aug 04 18:29:21.718568 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:29:21.718531    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6d5c87f9b9-5zl75"
Aug 04 18:29:21.735865 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:29:21.735789    2583 generic.go:356] "Generic (PLEG): container finished" podID="beb61a8e-02b7-48a8-bdef-5675f4095424" containerID="8d0be2ac640ff396fe5a115221c2b9ffb7547c54106045eff93a1e9f4b11db28" exitCode=0
Aug 04 18:29:21.735865 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:29:21.735828    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6d5c87f9b9-5zl75" event={"ID":"beb61a8e-02b7-48a8-bdef-5675f4095424","Type":"ContainerDied","Data":"8d0be2ac640ff396fe5a115221c2b9ffb7547c54106045eff93a1e9f4b11db28"}
Aug 04 18:29:21.735865 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:29:21.735853    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6d5c87f9b9-5zl75"
Aug 04 18:29:21.736116 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:29:21.735883    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6d5c87f9b9-5zl75" event={"ID":"beb61a8e-02b7-48a8-bdef-5675f4095424","Type":"ContainerDied","Data":"b562fb037ba21472dd9f8bf4f9a9e9f6acfbf58653aaabc3adb34bdedc032abd"}
Aug 04 18:29:21.736116 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:29:21.735917    2583 scope.go:117] "RemoveContainer" containerID="8d0be2ac640ff396fe5a115221c2b9ffb7547c54106045eff93a1e9f4b11db28"
Aug 04 18:29:21.794107 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:29:21.794081    2583 scope.go:117] "RemoveContainer" containerID="9291f6c5a81b8e84ea6ccfe9b04f264c95bf98fdbf2965b35566ab8c3f8f8cf7"
Aug 04 18:29:21.802243 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:29:21.802165    2583 scope.go:117] "RemoveContainer" containerID="8d0be2ac640ff396fe5a115221c2b9ffb7547c54106045eff93a1e9f4b11db28"
Aug 04 18:29:21.802493 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:29:21.802439    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d0be2ac640ff396fe5a115221c2b9ffb7547c54106045eff93a1e9f4b11db28\": container with ID starting with 8d0be2ac640ff396fe5a115221c2b9ffb7547c54106045eff93a1e9f4b11db28 not found: ID does not exist" containerID="8d0be2ac640ff396fe5a115221c2b9ffb7547c54106045eff93a1e9f4b11db28"
Aug 04 18:29:21.802598 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:29:21.802499    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d0be2ac640ff396fe5a115221c2b9ffb7547c54106045eff93a1e9f4b11db28"} err="failed to get container status \"8d0be2ac640ff396fe5a115221c2b9ffb7547c54106045eff93a1e9f4b11db28\": rpc error: code = NotFound desc = could not find container \"8d0be2ac640ff396fe5a115221c2b9ffb7547c54106045eff93a1e9f4b11db28\": container with ID starting with 8d0be2ac640ff396fe5a115221c2b9ffb7547c54106045eff93a1e9f4b11db28 not found: ID does not exist"
Aug 04 18:29:21.802598 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:29:21.802525    2583 scope.go:117] "RemoveContainer" containerID="9291f6c5a81b8e84ea6ccfe9b04f264c95bf98fdbf2965b35566ab8c3f8f8cf7"
Aug 04 18:29:21.802827 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:29:21.802797    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9291f6c5a81b8e84ea6ccfe9b04f264c95bf98fdbf2965b35566ab8c3f8f8cf7\": container with ID starting with 9291f6c5a81b8e84ea6ccfe9b04f264c95bf98fdbf2965b35566ab8c3f8f8cf7 not found: ID does not exist" containerID="9291f6c5a81b8e84ea6ccfe9b04f264c95bf98fdbf2965b35566ab8c3f8f8cf7"
Aug 04 18:29:21.802907 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:29:21.802826    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9291f6c5a81b8e84ea6ccfe9b04f264c95bf98fdbf2965b35566ab8c3f8f8cf7"} err="failed to get container status \"9291f6c5a81b8e84ea6ccfe9b04f264c95bf98fdbf2965b35566ab8c3f8f8cf7\": rpc error: code = NotFound desc = could not find container \"9291f6c5a81b8e84ea6ccfe9b04f264c95bf98fdbf2965b35566ab8c3f8f8cf7\": container with ID starting with 9291f6c5a81b8e84ea6ccfe9b04f264c95bf98fdbf2965b35566ab8c3f8f8cf7 not found: ID does not exist"
Aug 04 18:29:21.840762 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:29:21.840740    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/beb61a8e-02b7-48a8-bdef-5675f4095424-cabundle-cert\") pod \"beb61a8e-02b7-48a8-bdef-5675f4095424\" (UID: \"beb61a8e-02b7-48a8-bdef-5675f4095424\") "
Aug 04 18:29:21.840896 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:29:21.840785    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/beb61a8e-02b7-48a8-bdef-5675f4095424-dshm\") pod \"beb61a8e-02b7-48a8-bdef-5675f4095424\" (UID: \"beb61a8e-02b7-48a8-bdef-5675f4095424\") "
Aug 04 18:29:21.840896 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:29:21.840816    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/beb61a8e-02b7-48a8-bdef-5675f4095424-tmp-dir\") pod \"beb61a8e-02b7-48a8-bdef-5675f4095424\" (UID: \"beb61a8e-02b7-48a8-bdef-5675f4095424\") "
Aug 04 18:29:21.840896 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:29:21.840847    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/beb61a8e-02b7-48a8-bdef-5675f4095424-tls-certs\") pod \"beb61a8e-02b7-48a8-bdef-5675f4095424\" (UID: \"beb61a8e-02b7-48a8-bdef-5675f4095424\") "
Aug 04 18:29:21.840896 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:29:21.840895    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/beb61a8e-02b7-48a8-bdef-5675f4095424-home\") pod \"beb61a8e-02b7-48a8-bdef-5675f4095424\" (UID: \"beb61a8e-02b7-48a8-bdef-5675f4095424\") "
Aug 04 18:29:21.841122 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:29:21.840934    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/beb61a8e-02b7-48a8-bdef-5675f4095424-kserve-provision-location\") pod \"beb61a8e-02b7-48a8-bdef-5675f4095424\" (UID: \"beb61a8e-02b7-48a8-bdef-5675f4095424\") "
Aug 04 18:29:21.841122 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:29:21.840984    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/beb61a8e-02b7-48a8-bdef-5675f4095424-model-cache\") pod \"beb61a8e-02b7-48a8-bdef-5675f4095424\" (UID: \"beb61a8e-02b7-48a8-bdef-5675f4095424\") "
Aug 04 18:29:21.841122 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:29:21.841013    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l79n4\" (UniqueName: \"kubernetes.io/projected/beb61a8e-02b7-48a8-bdef-5675f4095424-kube-api-access-l79n4\") pod \"beb61a8e-02b7-48a8-bdef-5675f4095424\" (UID: \"beb61a8e-02b7-48a8-bdef-5675f4095424\") "
Aug 04 18:29:21.841280 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:29:21.841169    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/beb61a8e-02b7-48a8-bdef-5675f4095424-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "beb61a8e-02b7-48a8-bdef-5675f4095424" (UID: "beb61a8e-02b7-48a8-bdef-5675f4095424"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:29:21.841336 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:29:21.841282    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/beb61a8e-02b7-48a8-bdef-5675f4095424-model-cache" (OuterVolumeSpecName: "model-cache") pod "beb61a8e-02b7-48a8-bdef-5675f4095424" (UID: "beb61a8e-02b7-48a8-bdef-5675f4095424"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:29:21.841395 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:29:21.841375    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/beb61a8e-02b7-48a8-bdef-5675f4095424-model-cache\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:29:21.841451 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:29:21.841393    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/beb61a8e-02b7-48a8-bdef-5675f4095424-cabundle-cert\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:29:21.841527 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:29:21.841491    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/beb61a8e-02b7-48a8-bdef-5675f4095424-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "beb61a8e-02b7-48a8-bdef-5675f4095424" (UID: "beb61a8e-02b7-48a8-bdef-5675f4095424"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:29:21.841527 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:29:21.841508    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/beb61a8e-02b7-48a8-bdef-5675f4095424-home" (OuterVolumeSpecName: "home") pod "beb61a8e-02b7-48a8-bdef-5675f4095424" (UID: "beb61a8e-02b7-48a8-bdef-5675f4095424"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:29:21.846560 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:29:21.843826    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/beb61a8e-02b7-48a8-bdef-5675f4095424-kube-api-access-l79n4" (OuterVolumeSpecName: "kube-api-access-l79n4") pod "beb61a8e-02b7-48a8-bdef-5675f4095424" (UID: "beb61a8e-02b7-48a8-bdef-5675f4095424"). InnerVolumeSpecName "kube-api-access-l79n4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:29:21.846560 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:29:21.845340    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/beb61a8e-02b7-48a8-bdef-5675f4095424-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "beb61a8e-02b7-48a8-bdef-5675f4095424" (UID: "beb61a8e-02b7-48a8-bdef-5675f4095424"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:29:21.847054 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:29:21.847021    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/beb61a8e-02b7-48a8-bdef-5675f4095424-dshm" (OuterVolumeSpecName: "dshm") pod "beb61a8e-02b7-48a8-bdef-5675f4095424" (UID: "beb61a8e-02b7-48a8-bdef-5675f4095424"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:29:21.872561 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:29:21.872522    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/beb61a8e-02b7-48a8-bdef-5675f4095424-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "beb61a8e-02b7-48a8-bdef-5675f4095424" (UID: "beb61a8e-02b7-48a8-bdef-5675f4095424"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:29:21.914826 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:29:21.914803    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-56469b8ffc-nr4f4"
Aug 04 18:29:21.942582 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:29:21.942554    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/beb61a8e-02b7-48a8-bdef-5675f4095424-dshm\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:29:21.942733 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:29:21.942604    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/beb61a8e-02b7-48a8-bdef-5675f4095424-tmp-dir\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:29:21.942733 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:29:21.942619    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/beb61a8e-02b7-48a8-bdef-5675f4095424-tls-certs\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:29:21.942733 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:29:21.942630    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/beb61a8e-02b7-48a8-bdef-5675f4095424-home\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:29:21.942733 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:29:21.942639    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/beb61a8e-02b7-48a8-bdef-5675f4095424-kserve-provision-location\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:29:21.942733 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:29:21.942648    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l79n4\" (UniqueName: \"kubernetes.io/projected/beb61a8e-02b7-48a8-bdef-5675f4095424-kube-api-access-l79n4\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:29:22.043085 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:29:22.043053    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/813549a1-df99-46b7-a304-8b57329fdee8-tls-certs\") pod \"813549a1-df99-46b7-a304-8b57329fdee8\" (UID: \"813549a1-df99-46b7-a304-8b57329fdee8\") "
Aug 04 18:29:22.043085 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:29:22.043087    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/813549a1-df99-46b7-a304-8b57329fdee8-cabundle-cert\") pod \"813549a1-df99-46b7-a304-8b57329fdee8\" (UID: \"813549a1-df99-46b7-a304-8b57329fdee8\") "
Aug 04 18:29:22.043306 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:29:22.043111    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/813549a1-df99-46b7-a304-8b57329fdee8-tmp-dir\") pod \"813549a1-df99-46b7-a304-8b57329fdee8\" (UID: \"813549a1-df99-46b7-a304-8b57329fdee8\") "
Aug 04 18:29:22.043306 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:29:22.043157    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/813549a1-df99-46b7-a304-8b57329fdee8-kserve-provision-location\") pod \"813549a1-df99-46b7-a304-8b57329fdee8\" (UID: \"813549a1-df99-46b7-a304-8b57329fdee8\") "
Aug 04 18:29:22.043522 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:29:22.043490    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/813549a1-df99-46b7-a304-8b57329fdee8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "813549a1-df99-46b7-a304-8b57329fdee8" (UID: "813549a1-df99-46b7-a304-8b57329fdee8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:29:22.043636 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:29:22.043518    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/813549a1-df99-46b7-a304-8b57329fdee8-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "813549a1-df99-46b7-a304-8b57329fdee8" (UID: "813549a1-df99-46b7-a304-8b57329fdee8"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:29:22.043824 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:29:22.043803    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/813549a1-df99-46b7-a304-8b57329fdee8-cabundle-cert\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:29:22.043921 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:29:22.043827    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/813549a1-df99-46b7-a304-8b57329fdee8-tmp-dir\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:29:22.045618 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:29:22.045596    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/813549a1-df99-46b7-a304-8b57329fdee8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "813549a1-df99-46b7-a304-8b57329fdee8" (UID: "813549a1-df99-46b7-a304-8b57329fdee8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:29:22.061831 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:29:22.061807    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6d5c87f9b9-5zl75"]
Aug 04 18:29:22.068231 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:29:22.068208    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6d5c87f9b9-5zl75"]
Aug 04 18:29:22.069422 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:29:22.069402    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/813549a1-df99-46b7-a304-8b57329fdee8-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "813549a1-df99-46b7-a304-8b57329fdee8" (UID: "813549a1-df99-46b7-a304-8b57329fdee8"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:29:22.144376 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:29:22.144354    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/813549a1-df99-46b7-a304-8b57329fdee8-kserve-provision-location\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:29:22.144376 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:29:22.144375    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/813549a1-df99-46b7-a304-8b57329fdee8-tls-certs\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:29:22.743696 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:29:22.743609    2583 generic.go:356] "Generic (PLEG): container finished" podID="813549a1-df99-46b7-a304-8b57329fdee8" containerID="520ac651bce20e9a6a4b87315c8d76af3f900fc7e1242f0e0d0155bd6e46ba3e" exitCode=0
Aug 04 18:29:22.743696 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:29:22.743675    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-56469b8ffc-nr4f4"
Aug 04 18:29:22.744190 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:29:22.743735    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-56469b8ffc-nr4f4" event={"ID":"813549a1-df99-46b7-a304-8b57329fdee8","Type":"ContainerDied","Data":"520ac651bce20e9a6a4b87315c8d76af3f900fc7e1242f0e0d0155bd6e46ba3e"}
Aug 04 18:29:22.744190 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:29:22.743794    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-56469b8ffc-nr4f4" event={"ID":"813549a1-df99-46b7-a304-8b57329fdee8","Type":"ContainerDied","Data":"eab47b9266f6e527b2d0d9490c3c55c61279846c210fcf8f2f3c73cffb8786e8"}
Aug 04 18:29:22.744190 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:29:22.743825    2583 scope.go:117] "RemoveContainer" containerID="520ac651bce20e9a6a4b87315c8d76af3f900fc7e1242f0e0d0155bd6e46ba3e"
Aug 04 18:29:22.753363 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:29:22.753345    2583 scope.go:117] "RemoveContainer" containerID="fd2cff1f6bf8fc2fe8430147b2d9844594d00e3eee38286b991e7e89eaf5299a"
Aug 04 18:29:22.761539 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:29:22.761520    2583 scope.go:117] "RemoveContainer" containerID="520ac651bce20e9a6a4b87315c8d76af3f900fc7e1242f0e0d0155bd6e46ba3e"
Aug 04 18:29:22.761782 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:29:22.761763    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"520ac651bce20e9a6a4b87315c8d76af3f900fc7e1242f0e0d0155bd6e46ba3e\": container with ID starting with 520ac651bce20e9a6a4b87315c8d76af3f900fc7e1242f0e0d0155bd6e46ba3e not found: ID does not exist" containerID="520ac651bce20e9a6a4b87315c8d76af3f900fc7e1242f0e0d0155bd6e46ba3e"
Aug 04 18:29:22.761973 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:29:22.761790    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"520ac651bce20e9a6a4b87315c8d76af3f900fc7e1242f0e0d0155bd6e46ba3e"} err="failed to get container status \"520ac651bce20e9a6a4b87315c8d76af3f900fc7e1242f0e0d0155bd6e46ba3e\": rpc error: code = NotFound desc = could not find container \"520ac651bce20e9a6a4b87315c8d76af3f900fc7e1242f0e0d0155bd6e46ba3e\": container with ID starting with 520ac651bce20e9a6a4b87315c8d76af3f900fc7e1242f0e0d0155bd6e46ba3e not found: ID does not exist"
Aug 04 18:29:22.761973 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:29:22.761807    2583 scope.go:117] "RemoveContainer" containerID="fd2cff1f6bf8fc2fe8430147b2d9844594d00e3eee38286b991e7e89eaf5299a"
Aug 04 18:29:22.762074 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:29:22.762050    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd2cff1f6bf8fc2fe8430147b2d9844594d00e3eee38286b991e7e89eaf5299a\": container with ID starting with fd2cff1f6bf8fc2fe8430147b2d9844594d00e3eee38286b991e7e89eaf5299a not found: ID does not exist" containerID="fd2cff1f6bf8fc2fe8430147b2d9844594d00e3eee38286b991e7e89eaf5299a"
Aug 04 18:29:22.762111 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:29:22.762072    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd2cff1f6bf8fc2fe8430147b2d9844594d00e3eee38286b991e7e89eaf5299a"} err="failed to get container status \"fd2cff1f6bf8fc2fe8430147b2d9844594d00e3eee38286b991e7e89eaf5299a\": rpc error: code = NotFound desc = could not find container \"fd2cff1f6bf8fc2fe8430147b2d9844594d00e3eee38286b991e7e89eaf5299a\": container with ID starting with fd2cff1f6bf8fc2fe8430147b2d9844594d00e3eee38286b991e7e89eaf5299a not found: ID does not exist"
Aug 04 18:29:22.778762 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:29:22.778740    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-56469b8ffc-nr4f4"]
Aug 04 18:29:22.781399 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:29:22.781379    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-56469b8ffc-nr4f4"]
Aug 04 18:29:22.942556 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:29:22.942525    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="813549a1-df99-46b7-a304-8b57329fdee8" path="/var/lib/kubelet/pods/813549a1-df99-46b7-a304-8b57329fdee8/volumes"
Aug 04 18:29:22.942990 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:29:22.942978    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="beb61a8e-02b7-48a8-bdef-5675f4095424" path="/var/lib/kubelet/pods/beb61a8e-02b7-48a8-bdef-5675f4095424/volumes"
Aug 04 18:29:24.609247 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:29:24.609212    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-2hzr2"]
Aug 04 18:29:34.027144 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:29:34.027111    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-7sh4p"]
Aug 04 18:29:34.027517 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:29:34.027464    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="beb61a8e-02b7-48a8-bdef-5675f4095424" containerName="main"
Aug 04 18:29:34.027517 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:29:34.027494    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="beb61a8e-02b7-48a8-bdef-5675f4095424" containerName="main"
Aug 04 18:29:34.027608 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:29:34.027525    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="813549a1-df99-46b7-a304-8b57329fdee8" containerName="storage-initializer"
Aug 04 18:29:34.027608 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:29:34.027534    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="813549a1-df99-46b7-a304-8b57329fdee8" containerName="storage-initializer"
Aug 04 18:29:34.027608 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:29:34.027550    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="813549a1-df99-46b7-a304-8b57329fdee8" containerName="main"
Aug 04 18:29:34.027608 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:29:34.027556    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="813549a1-df99-46b7-a304-8b57329fdee8" containerName="main"
Aug 04 18:29:34.027608 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:29:34.027568    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="beb61a8e-02b7-48a8-bdef-5675f4095424" containerName="storage-initializer"
Aug 04 18:29:34.027608 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:29:34.027573    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="beb61a8e-02b7-48a8-bdef-5675f4095424" containerName="storage-initializer"
Aug 04 18:29:34.027788 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:29:34.027626    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="beb61a8e-02b7-48a8-bdef-5675f4095424" containerName="main"
Aug 04 18:29:34.027788 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:29:34.027638    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="813549a1-df99-46b7-a304-8b57329fdee8" containerName="main"
Aug 04 18:29:34.030621 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:29:34.030602    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-7sh4p"
Aug 04 18:29:34.034985 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:29:34.034958    2583 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"default-dockercfg-59ldg\" is forbidden: User \"system:node:ip-10-0-135-124.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"e2e-test-force-stop-3e394e04\": no relationship found between node 'ip-10-0-135-124.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"e2e-test-force-stop-3e394e04\"/\"default-dockercfg-59ldg\"" type="*v1.Secret"
Aug 04 18:29:34.034985 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:29:34.034959    2583 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"stop-v1-kserve-self-signed-certs\" is forbidden: User \"system:node:ip-10-0-135-124.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"e2e-test-force-stop-3e394e04\": no relationship found between node 'ip-10-0-135-124.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"e2e-test-force-stop-3e394e04\"/\"stop-v1-kserve-self-signed-certs\"" type="*v1.Secret"
Aug 04 18:29:34.035624 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:29:34.035598    2583 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"kube-root-ca.crt\" is forbidden: User \"system:node:ip-10-0-135-124.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"e2e-test-force-stop-3e394e04\": no relationship found between node 'ip-10-0-135-124.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"e2e-test-force-stop-3e394e04\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap"
Aug 04 18:29:34.036133 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:29:34.036107    2583 status_manager.go:1018] "Failed to get status for pod" err="pods \"stop-v1-kserve-5657b88c95-7sh4p\" is forbidden: User \"system:node:ip-10-0-135-124.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"e2e-test-force-stop-3e394e04\": no relationship found between node 'ip-10-0-135-124.ec2.internal' and this object" podUID="3fe52df4-b3cc-4ad6-bcf0-5938fb053133" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-7sh4p"
Aug 04 18:29:34.042311 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:29:34.042287    2583 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"openshift-service-ca.crt\" is forbidden: User \"system:node:ip-10-0-135-124.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"e2e-test-force-stop-3e394e04\": no relationship found between node 'ip-10-0-135-124.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"e2e-test-force-stop-3e394e04\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap"
Aug 04 18:29:34.058733 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:29:34.058710    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-7sh4p"]
Aug 04 18:29:34.140016 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:29:34.139984    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3fe52df4-b3cc-4ad6-bcf0-5938fb053133-model-cache\") pod \"stop-v1-kserve-5657b88c95-7sh4p\" (UID: \"3fe52df4-b3cc-4ad6-bcf0-5938fb053133\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-7sh4p"
Aug 04 18:29:34.140016 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:29:34.140017    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3fe52df4-b3cc-4ad6-bcf0-5938fb053133-tls-certs\") pod \"stop-v1-kserve-5657b88c95-7sh4p\" (UID: \"3fe52df4-b3cc-4ad6-bcf0-5938fb053133\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-7sh4p"
Aug 04 18:29:34.140182 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:29:34.140036    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3fe52df4-b3cc-4ad6-bcf0-5938fb053133-dshm\") pod \"stop-v1-kserve-5657b88c95-7sh4p\" (UID: \"3fe52df4-b3cc-4ad6-bcf0-5938fb053133\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-7sh4p"
Aug 04 18:29:34.140182 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:29:34.140108    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3fe52df4-b3cc-4ad6-bcf0-5938fb053133-home\") pod \"stop-v1-kserve-5657b88c95-7sh4p\" (UID: \"3fe52df4-b3cc-4ad6-bcf0-5938fb053133\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-7sh4p"
Aug 04 18:29:34.140182 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:29:34.140137    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3fe52df4-b3cc-4ad6-bcf0-5938fb053133-tmp-dir\") pod \"stop-v1-kserve-5657b88c95-7sh4p\" (UID: \"3fe52df4-b3cc-4ad6-bcf0-5938fb053133\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-7sh4p"
Aug 04 18:29:34.140286 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:29:34.140176    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kb6pp\" (UniqueName: \"kubernetes.io/projected/3fe52df4-b3cc-4ad6-bcf0-5938fb053133-kube-api-access-kb6pp\") pod \"stop-v1-kserve-5657b88c95-7sh4p\" (UID: \"3fe52df4-b3cc-4ad6-bcf0-5938fb053133\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-7sh4p"
Aug 04 18:29:34.241256 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:29:34.241216    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3fe52df4-b3cc-4ad6-bcf0-5938fb053133-tls-certs\") pod \"stop-v1-kserve-5657b88c95-7sh4p\" (UID: \"3fe52df4-b3cc-4ad6-bcf0-5938fb053133\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-7sh4p"
Aug 04 18:29:34.241425 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:29:34.241266    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3fe52df4-b3cc-4ad6-bcf0-5938fb053133-dshm\") pod \"stop-v1-kserve-5657b88c95-7sh4p\" (UID: \"3fe52df4-b3cc-4ad6-bcf0-5938fb053133\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-7sh4p"
Aug 04 18:29:34.241425 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:29:34.241318    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3fe52df4-b3cc-4ad6-bcf0-5938fb053133-home\") pod \"stop-v1-kserve-5657b88c95-7sh4p\" (UID: \"3fe52df4-b3cc-4ad6-bcf0-5938fb053133\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-7sh4p"
Aug 04 18:29:34.241425 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:29:34.241349    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3fe52df4-b3cc-4ad6-bcf0-5938fb053133-tmp-dir\") pod \"stop-v1-kserve-5657b88c95-7sh4p\" (UID: \"3fe52df4-b3cc-4ad6-bcf0-5938fb053133\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-7sh4p"
Aug 04 18:29:34.241425 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:29:34.241372    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kb6pp\" (UniqueName: \"kubernetes.io/projected/3fe52df4-b3cc-4ad6-bcf0-5938fb053133-kube-api-access-kb6pp\") pod \"stop-v1-kserve-5657b88c95-7sh4p\" (UID: \"3fe52df4-b3cc-4ad6-bcf0-5938fb053133\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-7sh4p"
Aug 04 18:29:34.241662 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:29:34.241518    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3fe52df4-b3cc-4ad6-bcf0-5938fb053133-model-cache\") pod \"stop-v1-kserve-5657b88c95-7sh4p\" (UID: \"3fe52df4-b3cc-4ad6-bcf0-5938fb053133\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-7sh4p"
Aug 04 18:29:34.241843 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:29:34.241813    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3fe52df4-b3cc-4ad6-bcf0-5938fb053133-home\") pod \"stop-v1-kserve-5657b88c95-7sh4p\" (UID: \"3fe52df4-b3cc-4ad6-bcf0-5938fb053133\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-7sh4p"
Aug 04 18:29:34.241932 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:29:34.241827    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3fe52df4-b3cc-4ad6-bcf0-5938fb053133-tmp-dir\") pod \"stop-v1-kserve-5657b88c95-7sh4p\" (UID: \"3fe52df4-b3cc-4ad6-bcf0-5938fb053133\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-7sh4p"
Aug 04 18:29:34.241932 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:29:34.241863    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3fe52df4-b3cc-4ad6-bcf0-5938fb053133-model-cache\") pod \"stop-v1-kserve-5657b88c95-7sh4p\" (UID: \"3fe52df4-b3cc-4ad6-bcf0-5938fb053133\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-7sh4p"
Aug 04 18:29:34.243715 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:29:34.243690    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3fe52df4-b3cc-4ad6-bcf0-5938fb053133-dshm\") pod \"stop-v1-kserve-5657b88c95-7sh4p\" (UID: \"3fe52df4-b3cc-4ad6-bcf0-5938fb053133\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-7sh4p"
Aug 04 18:29:35.004560 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:29:35.004522    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-force-stop-3e394e04\"/\"openshift-service-ca.crt\""
Aug 04 18:29:35.047287 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:29:35.047256    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-force-stop-3e394e04\"/\"kube-root-ca.crt\""
Aug 04 18:29:35.054064 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:29:35.054034    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kb6pp\" (UniqueName: \"kubernetes.io/projected/3fe52df4-b3cc-4ad6-bcf0-5938fb053133-kube-api-access-kb6pp\") pod \"stop-v1-kserve-5657b88c95-7sh4p\" (UID: \"3fe52df4-b3cc-4ad6-bcf0-5938fb053133\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-7sh4p"
Aug 04 18:29:35.080762 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:29:35.080737    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-force-stop-3e394e04\"/\"stop-v1-kserve-self-signed-certs\""
Aug 04 18:29:35.083996 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:29:35.083973    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3fe52df4-b3cc-4ad6-bcf0-5938fb053133-tls-certs\") pod \"stop-v1-kserve-5657b88c95-7sh4p\" (UID: \"3fe52df4-b3cc-4ad6-bcf0-5938fb053133\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-7sh4p"
Aug 04 18:29:35.571854 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:29:35.571818    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-force-stop-3e394e04\"/\"default-dockercfg-59ldg\""
Aug 04 18:29:35.581572 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:29:35.581552    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-7sh4p"
Aug 04 18:29:35.718879 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:29:35.718850    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-7sh4p"]
Aug 04 18:29:35.720342 ip-10-0-135-124 kubenswrapper[2583]: W0804 18:29:35.720315    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fe52df4_b3cc_4ad6_bcf0_5938fb053133.slice/crio-baf9a688b492f3130352a85779f532a9517d42e3a1e73670b27b452e352be01b WatchSource:0}: Error finding container baf9a688b492f3130352a85779f532a9517d42e3a1e73670b27b452e352be01b: Status 404 returned error can't find the container with id baf9a688b492f3130352a85779f532a9517d42e3a1e73670b27b452e352be01b
Aug 04 18:29:35.822351 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:29:35.822267    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-7sh4p" event={"ID":"3fe52df4-b3cc-4ad6-bcf0-5938fb053133","Type":"ContainerStarted","Data":"a8427e2bdb321b01563c649a7dff2c683fb1fe3f5333541c693be191f86ff9f4"}
Aug 04 18:29:35.822351 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:29:35.822329    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-7sh4p" event={"ID":"3fe52df4-b3cc-4ad6-bcf0-5938fb053133","Type":"ContainerStarted","Data":"baf9a688b492f3130352a85779f532a9517d42e3a1e73670b27b452e352be01b"}
Aug 04 18:29:35.840618 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:29:35.840563    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-7sh4p" podStartSLOduration=2.84054655 podStartE2EDuration="2.84054655s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:29:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:29:35.839123077 +0000 UTC m=+1789.420341656" watchObservedRunningTime="2026-08-04 18:29:35.84054655 +0000 UTC m=+1789.421765127"
Aug 04 18:29:36.581949 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:29:36.581917    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-7sh4p"
Aug 04 18:29:39.628763 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:29:39.628679    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-2hzr2" podUID="b4dd6117-515a-4eab-9975-83bb59758255" containerName="main" containerID="cri-o://c6d3e5715ce4d4fce9366f215cd397dcf6ee10caa7dcc668c98d8533356be279" gracePeriod=45
Aug 04 18:29:39.845295 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:29:39.845264    2583 generic.go:356] "Generic (PLEG): container finished" podID="b4dd6117-515a-4eab-9975-83bb59758255" containerID="c6d3e5715ce4d4fce9366f215cd397dcf6ee10caa7dcc668c98d8533356be279" exitCode=0
Aug 04 18:29:39.845501 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:29:39.845337    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-2hzr2" event={"ID":"b4dd6117-515a-4eab-9975-83bb59758255","Type":"ContainerDied","Data":"c6d3e5715ce4d4fce9366f215cd397dcf6ee10caa7dcc668c98d8533356be279"}
Aug 04 18:29:39.883369 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:29:39.883317    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-2hzr2"
Aug 04 18:29:39.992943 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:29:39.992910    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4dd6117-515a-4eab-9975-83bb59758255-model-cache\") pod \"b4dd6117-515a-4eab-9975-83bb59758255\" (UID: \"b4dd6117-515a-4eab-9975-83bb59758255\") "
Aug 04 18:29:39.992943 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:29:39.992948    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4dd6117-515a-4eab-9975-83bb59758255-tls-certs\") pod \"b4dd6117-515a-4eab-9975-83bb59758255\" (UID: \"b4dd6117-515a-4eab-9975-83bb59758255\") "
Aug 04 18:29:39.993189 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:29:39.992970    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hbxqx\" (UniqueName: \"kubernetes.io/projected/b4dd6117-515a-4eab-9975-83bb59758255-kube-api-access-hbxqx\") pod \"b4dd6117-515a-4eab-9975-83bb59758255\" (UID: \"b4dd6117-515a-4eab-9975-83bb59758255\") "
Aug 04 18:29:39.993189 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:29:39.993027    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4dd6117-515a-4eab-9975-83bb59758255-dshm\") pod \"b4dd6117-515a-4eab-9975-83bb59758255\" (UID: \"b4dd6117-515a-4eab-9975-83bb59758255\") "
Aug 04 18:29:39.993189 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:29:39.993058    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4dd6117-515a-4eab-9975-83bb59758255-tmp-dir\") pod \"b4dd6117-515a-4eab-9975-83bb59758255\" (UID: \"b4dd6117-515a-4eab-9975-83bb59758255\") "
Aug 04 18:29:39.993189 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:29:39.993076    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b4dd6117-515a-4eab-9975-83bb59758255-home\") pod \"b4dd6117-515a-4eab-9975-83bb59758255\" (UID: \"b4dd6117-515a-4eab-9975-83bb59758255\") "
Aug 04 18:29:39.993387 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:29:39.993238    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4dd6117-515a-4eab-9975-83bb59758255-model-cache" (OuterVolumeSpecName: "model-cache") pod "b4dd6117-515a-4eab-9975-83bb59758255" (UID: "b4dd6117-515a-4eab-9975-83bb59758255"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:29:39.993387 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:29:39.993328    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4dd6117-515a-4eab-9975-83bb59758255-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b4dd6117-515a-4eab-9975-83bb59758255" (UID: "b4dd6117-515a-4eab-9975-83bb59758255"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:29:39.993387 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:29:39.993361    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4dd6117-515a-4eab-9975-83bb59758255-model-cache\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:29:39.993543 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:29:39.993432    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4dd6117-515a-4eab-9975-83bb59758255-home" (OuterVolumeSpecName: "home") pod "b4dd6117-515a-4eab-9975-83bb59758255" (UID: "b4dd6117-515a-4eab-9975-83bb59758255"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:29:39.995265 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:29:39.995243    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4dd6117-515a-4eab-9975-83bb59758255-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b4dd6117-515a-4eab-9975-83bb59758255" (UID: "b4dd6117-515a-4eab-9975-83bb59758255"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:29:39.995265 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:29:39.995254    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4dd6117-515a-4eab-9975-83bb59758255-kube-api-access-hbxqx" (OuterVolumeSpecName: "kube-api-access-hbxqx") pod "b4dd6117-515a-4eab-9975-83bb59758255" (UID: "b4dd6117-515a-4eab-9975-83bb59758255"). InnerVolumeSpecName "kube-api-access-hbxqx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:29:39.995354 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:29:39.995303    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4dd6117-515a-4eab-9975-83bb59758255-dshm" (OuterVolumeSpecName: "dshm") pod "b4dd6117-515a-4eab-9975-83bb59758255" (UID: "b4dd6117-515a-4eab-9975-83bb59758255"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:29:40.094279 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:29:40.094238    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4dd6117-515a-4eab-9975-83bb59758255-tls-certs\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:29:40.094279 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:29:40.094272    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hbxqx\" (UniqueName: \"kubernetes.io/projected/b4dd6117-515a-4eab-9975-83bb59758255-kube-api-access-hbxqx\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:29:40.094279 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:29:40.094285    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4dd6117-515a-4eab-9975-83bb59758255-dshm\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:29:40.094555 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:29:40.094299    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4dd6117-515a-4eab-9975-83bb59758255-tmp-dir\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:29:40.094555 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:29:40.094313    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b4dd6117-515a-4eab-9975-83bb59758255-home\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:29:40.851874 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:29:40.851842    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-2hzr2"
Aug 04 18:29:40.851874 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:29:40.851855    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-2hzr2" event={"ID":"b4dd6117-515a-4eab-9975-83bb59758255","Type":"ContainerDied","Data":"0c2d3054bbeb8b68972f817a8c397513c4b9f950a7c23e15dfebc0bcf9747e95"}
Aug 04 18:29:40.852369 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:29:40.851942    2583 scope.go:117] "RemoveContainer" containerID="c6d3e5715ce4d4fce9366f215cd397dcf6ee10caa7dcc668c98d8533356be279"
Aug 04 18:29:40.877010 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:29:40.876980    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-2hzr2"]
Aug 04 18:29:40.881150 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:29:40.881127    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v1-kserve-fb566d667-2hzr2"]
Aug 04 18:29:40.941884 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:29:40.941852    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b4dd6117-515a-4eab-9975-83bb59758255" path="/var/lib/kubelet/pods/b4dd6117-515a-4eab-9975-83bb59758255/volumes"
Aug 04 18:29:45.582035 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:29:45.581997    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-7sh4p"
Aug 04 18:29:45.594828 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:29:45.594798    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-7sh4p"
Aug 04 18:29:45.890250 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:29:45.890168    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-7sh4p"
Aug 04 18:29:47.029608 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:29:47.029494    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-wskfb_6157c2d3-2fd0-4544-88ad-434ac75f8c04/console-operator/1.log"
Aug 04 18:29:47.043250 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:29:47.036188    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-wskfb_6157c2d3-2fd0-4544-88ad-434ac75f8c04/console-operator/1.log"
Aug 04 18:31:01.701902 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:31:01.701871    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-7sh4p"]
Aug 04 18:31:05.525033 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:31:05.524999    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-rtmbx"]
Aug 04 18:31:05.525397 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:31:05.525382    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4dd6117-515a-4eab-9975-83bb59758255" containerName="main"
Aug 04 18:31:05.525397 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:31:05.525393    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4dd6117-515a-4eab-9975-83bb59758255" containerName="main"
Aug 04 18:31:05.525489 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:31:05.525458    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="b4dd6117-515a-4eab-9975-83bb59758255" containerName="main"
Aug 04 18:31:05.527206 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:31:05.527189    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-rtmbx"
Aug 04 18:31:05.532159 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:31:05.532137    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-decommission-203cd6d1\"/\"default-dockercfg-sq7bp\""
Aug 04 18:31:05.533311 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:31:05.533290    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-decommission-203cd6d1\"/\"kube-root-ca.crt\""
Aug 04 18:31:05.535193 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:31:05.535173    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-decommission-203cd6d1\"/\"decom-v1-kserve-self-signed-certs\""
Aug 04 18:31:05.535301 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:31:05.535234    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-decommission-203cd6d1\"/\"openshift-service-ca.crt\""
Aug 04 18:31:05.543380 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:31:05.543356    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-rtmbx"]
Aug 04 18:31:05.607348 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:31:05.607316    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2fgxj\" (UniqueName: \"kubernetes.io/projected/4f5add6e-57c5-4940-b289-67ef3c98449c-kube-api-access-2fgxj\") pod \"decom-v1-kserve-6cb797fb75-rtmbx\" (UID: \"4f5add6e-57c5-4940-b289-67ef3c98449c\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-rtmbx"
Aug 04 18:31:05.607511 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:31:05.607368    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f5add6e-57c5-4940-b289-67ef3c98449c-tls-certs\") pod \"decom-v1-kserve-6cb797fb75-rtmbx\" (UID: \"4f5add6e-57c5-4940-b289-67ef3c98449c\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-rtmbx"
Aug 04 18:31:05.607511 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:31:05.607396    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f5add6e-57c5-4940-b289-67ef3c98449c-dshm\") pod \"decom-v1-kserve-6cb797fb75-rtmbx\" (UID: \"4f5add6e-57c5-4940-b289-67ef3c98449c\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-rtmbx"
Aug 04 18:31:05.607511 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:31:05.607416    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f5add6e-57c5-4940-b289-67ef3c98449c-tmp-dir\") pod \"decom-v1-kserve-6cb797fb75-rtmbx\" (UID: \"4f5add6e-57c5-4940-b289-67ef3c98449c\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-rtmbx"
Aug 04 18:31:05.607511 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:31:05.607444    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f5add6e-57c5-4940-b289-67ef3c98449c-home\") pod \"decom-v1-kserve-6cb797fb75-rtmbx\" (UID: \"4f5add6e-57c5-4940-b289-67ef3c98449c\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-rtmbx"
Aug 04 18:31:05.607733 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:31:05.607589    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f5add6e-57c5-4940-b289-67ef3c98449c-model-cache\") pod \"decom-v1-kserve-6cb797fb75-rtmbx\" (UID: \"4f5add6e-57c5-4940-b289-67ef3c98449c\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-rtmbx"
Aug 04 18:31:05.708664 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:31:05.708636    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f5add6e-57c5-4940-b289-67ef3c98449c-model-cache\") pod \"decom-v1-kserve-6cb797fb75-rtmbx\" (UID: \"4f5add6e-57c5-4940-b289-67ef3c98449c\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-rtmbx"
Aug 04 18:31:05.708784 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:31:05.708688    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fgxj\" (UniqueName: \"kubernetes.io/projected/4f5add6e-57c5-4940-b289-67ef3c98449c-kube-api-access-2fgxj\") pod \"decom-v1-kserve-6cb797fb75-rtmbx\" (UID: \"4f5add6e-57c5-4940-b289-67ef3c98449c\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-rtmbx"
Aug 04 18:31:05.708784 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:31:05.708722    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f5add6e-57c5-4940-b289-67ef3c98449c-tls-certs\") pod \"decom-v1-kserve-6cb797fb75-rtmbx\" (UID: \"4f5add6e-57c5-4940-b289-67ef3c98449c\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-rtmbx"
Aug 04 18:31:05.708784 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:31:05.708751    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f5add6e-57c5-4940-b289-67ef3c98449c-dshm\") pod \"decom-v1-kserve-6cb797fb75-rtmbx\" (UID: \"4f5add6e-57c5-4940-b289-67ef3c98449c\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-rtmbx"
Aug 04 18:31:05.708784 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:31:05.708772    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f5add6e-57c5-4940-b289-67ef3c98449c-tmp-dir\") pod \"decom-v1-kserve-6cb797fb75-rtmbx\" (UID: \"4f5add6e-57c5-4940-b289-67ef3c98449c\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-rtmbx"
Aug 04 18:31:05.709010 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:31:05.708800    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f5add6e-57c5-4940-b289-67ef3c98449c-home\") pod \"decom-v1-kserve-6cb797fb75-rtmbx\" (UID: \"4f5add6e-57c5-4940-b289-67ef3c98449c\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-rtmbx"
Aug 04 18:31:05.709151 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:31:05.709115    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f5add6e-57c5-4940-b289-67ef3c98449c-model-cache\") pod \"decom-v1-kserve-6cb797fb75-rtmbx\" (UID: \"4f5add6e-57c5-4940-b289-67ef3c98449c\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-rtmbx"
Aug 04 18:31:05.709236 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:31:05.709160    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f5add6e-57c5-4940-b289-67ef3c98449c-home\") pod \"decom-v1-kserve-6cb797fb75-rtmbx\" (UID: \"4f5add6e-57c5-4940-b289-67ef3c98449c\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-rtmbx"
Aug 04 18:31:05.709236 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:31:05.709220    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f5add6e-57c5-4940-b289-67ef3c98449c-tmp-dir\") pod \"decom-v1-kserve-6cb797fb75-rtmbx\" (UID: \"4f5add6e-57c5-4940-b289-67ef3c98449c\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-rtmbx"
Aug 04 18:31:05.710989 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:31:05.710970    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f5add6e-57c5-4940-b289-67ef3c98449c-dshm\") pod \"decom-v1-kserve-6cb797fb75-rtmbx\" (UID: \"4f5add6e-57c5-4940-b289-67ef3c98449c\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-rtmbx"
Aug 04 18:31:05.711318 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:31:05.711301    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f5add6e-57c5-4940-b289-67ef3c98449c-tls-certs\") pod \"decom-v1-kserve-6cb797fb75-rtmbx\" (UID: \"4f5add6e-57c5-4940-b289-67ef3c98449c\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-rtmbx"
Aug 04 18:31:05.717970 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:31:05.717943    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fgxj\" (UniqueName: \"kubernetes.io/projected/4f5add6e-57c5-4940-b289-67ef3c98449c-kube-api-access-2fgxj\") pod \"decom-v1-kserve-6cb797fb75-rtmbx\" (UID: \"4f5add6e-57c5-4940-b289-67ef3c98449c\") " pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-rtmbx"
Aug 04 18:31:05.837422 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:31:05.837360    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-rtmbx"
Aug 04 18:31:06.003914 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:31:06.003830    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-rtmbx"]
Aug 04 18:31:06.006300 ip-10-0-135-124 kubenswrapper[2583]: W0804 18:31:06.006268    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f5add6e_57c5_4940_b289_67ef3c98449c.slice/crio-63c9bea715ac341e9a3c8a786b9cf90f34d0035890197cbd4cd72c03f8df41ca WatchSource:0}: Error finding container 63c9bea715ac341e9a3c8a786b9cf90f34d0035890197cbd4cd72c03f8df41ca: Status 404 returned error can't find the container with id 63c9bea715ac341e9a3c8a786b9cf90f34d0035890197cbd4cd72c03f8df41ca
Aug 04 18:31:06.307129 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:31:06.307087    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-rtmbx" event={"ID":"4f5add6e-57c5-4940-b289-67ef3c98449c","Type":"ContainerStarted","Data":"a840d83698b4038a35a38d99ccca94757ced22af2a81c1ad086cec4a33e25e94"}
Aug 04 18:31:06.307289 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:31:06.307154    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-rtmbx" event={"ID":"4f5add6e-57c5-4940-b289-67ef3c98449c","Type":"ContainerStarted","Data":"63c9bea715ac341e9a3c8a786b9cf90f34d0035890197cbd4cd72c03f8df41ca"}
Aug 04 18:31:06.332403 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:31:06.332335    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-rtmbx" podStartSLOduration=1.33231509 podStartE2EDuration="1.33231509s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:31:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:31:06.329378709 +0000 UTC m=+1879.910597288" watchObservedRunningTime="2026-08-04 18:31:06.33231509 +0000 UTC m=+1879.913533669"
Aug 04 18:31:06.838295 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:31:06.838267    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-rtmbx"
Aug 04 18:31:15.838538 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:31:15.838409    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-rtmbx"
Aug 04 18:31:15.850817 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:31:15.850789    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-rtmbx"
Aug 04 18:31:16.369683 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:31:16.369650    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-rtmbx"
Aug 04 18:31:16.720842 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:31:16.720806    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-7sh4p" podUID="3fe52df4-b3cc-4ad6-bcf0-5938fb053133" containerName="main" containerID="cri-o://a8427e2bdb321b01563c649a7dff2c683fb1fe3f5333541c693be191f86ff9f4" gracePeriod=45
Aug 04 18:31:16.880263 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:31:16.880224    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-7sh4p" podUID="3fe52df4-b3cc-4ad6-bcf0-5938fb053133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": dial tcp 10.132.0.66:8000: connect: connection refused"
Aug 04 18:31:16.974234 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:31:16.974178    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-7sh4p"
Aug 04 18:31:17.105652 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:31:17.105618    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3fe52df4-b3cc-4ad6-bcf0-5938fb053133-model-cache\") pod \"3fe52df4-b3cc-4ad6-bcf0-5938fb053133\" (UID: \"3fe52df4-b3cc-4ad6-bcf0-5938fb053133\") "
Aug 04 18:31:17.105816 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:31:17.105684    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3fe52df4-b3cc-4ad6-bcf0-5938fb053133-tmp-dir\") pod \"3fe52df4-b3cc-4ad6-bcf0-5938fb053133\" (UID: \"3fe52df4-b3cc-4ad6-bcf0-5938fb053133\") "
Aug 04 18:31:17.105816 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:31:17.105744    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3fe52df4-b3cc-4ad6-bcf0-5938fb053133-home\") pod \"3fe52df4-b3cc-4ad6-bcf0-5938fb053133\" (UID: \"3fe52df4-b3cc-4ad6-bcf0-5938fb053133\") "
Aug 04 18:31:17.105816 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:31:17.105789    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3fe52df4-b3cc-4ad6-bcf0-5938fb053133-dshm\") pod \"3fe52df4-b3cc-4ad6-bcf0-5938fb053133\" (UID: \"3fe52df4-b3cc-4ad6-bcf0-5938fb053133\") "
Aug 04 18:31:17.105996 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:31:17.105821    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kb6pp\" (UniqueName: \"kubernetes.io/projected/3fe52df4-b3cc-4ad6-bcf0-5938fb053133-kube-api-access-kb6pp\") pod \"3fe52df4-b3cc-4ad6-bcf0-5938fb053133\" (UID: \"3fe52df4-b3cc-4ad6-bcf0-5938fb053133\") "
Aug 04 18:31:17.105996 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:31:17.105846    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3fe52df4-b3cc-4ad6-bcf0-5938fb053133-tls-certs\") pod \"3fe52df4-b3cc-4ad6-bcf0-5938fb053133\" (UID: \"3fe52df4-b3cc-4ad6-bcf0-5938fb053133\") "
Aug 04 18:31:17.105996 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:31:17.105895    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fe52df4-b3cc-4ad6-bcf0-5938fb053133-model-cache" (OuterVolumeSpecName: "model-cache") pod "3fe52df4-b3cc-4ad6-bcf0-5938fb053133" (UID: "3fe52df4-b3cc-4ad6-bcf0-5938fb053133"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:31:17.105996 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:31:17.105912    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fe52df4-b3cc-4ad6-bcf0-5938fb053133-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3fe52df4-b3cc-4ad6-bcf0-5938fb053133" (UID: "3fe52df4-b3cc-4ad6-bcf0-5938fb053133"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:31:17.105996 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:31:17.105963    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fe52df4-b3cc-4ad6-bcf0-5938fb053133-home" (OuterVolumeSpecName: "home") pod "3fe52df4-b3cc-4ad6-bcf0-5938fb053133" (UID: "3fe52df4-b3cc-4ad6-bcf0-5938fb053133"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:31:17.106272 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:31:17.106151    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3fe52df4-b3cc-4ad6-bcf0-5938fb053133-model-cache\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:31:17.106272 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:31:17.106172    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3fe52df4-b3cc-4ad6-bcf0-5938fb053133-tmp-dir\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:31:17.106272 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:31:17.106181    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3fe52df4-b3cc-4ad6-bcf0-5938fb053133-home\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:31:17.108037 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:31:17.108013    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fe52df4-b3cc-4ad6-bcf0-5938fb053133-dshm" (OuterVolumeSpecName: "dshm") pod "3fe52df4-b3cc-4ad6-bcf0-5938fb053133" (UID: "3fe52df4-b3cc-4ad6-bcf0-5938fb053133"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:31:17.108436 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:31:17.108418    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3fe52df4-b3cc-4ad6-bcf0-5938fb053133-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3fe52df4-b3cc-4ad6-bcf0-5938fb053133" (UID: "3fe52df4-b3cc-4ad6-bcf0-5938fb053133"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:31:17.108537 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:31:17.108437    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fe52df4-b3cc-4ad6-bcf0-5938fb053133-kube-api-access-kb6pp" (OuterVolumeSpecName: "kube-api-access-kb6pp") pod "3fe52df4-b3cc-4ad6-bcf0-5938fb053133" (UID: "3fe52df4-b3cc-4ad6-bcf0-5938fb053133"). InnerVolumeSpecName "kube-api-access-kb6pp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:31:17.206925 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:31:17.206898    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3fe52df4-b3cc-4ad6-bcf0-5938fb053133-dshm\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:31:17.206925 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:31:17.206923    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kb6pp\" (UniqueName: \"kubernetes.io/projected/3fe52df4-b3cc-4ad6-bcf0-5938fb053133-kube-api-access-kb6pp\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:31:17.206925 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:31:17.206933    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3fe52df4-b3cc-4ad6-bcf0-5938fb053133-tls-certs\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:31:17.365303 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:31:17.365225    2583 generic.go:356] "Generic (PLEG): container finished" podID="3fe52df4-b3cc-4ad6-bcf0-5938fb053133" containerID="a8427e2bdb321b01563c649a7dff2c683fb1fe3f5333541c693be191f86ff9f4" exitCode=0
Aug 04 18:31:17.365303 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:31:17.365269    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-7sh4p"
Aug 04 18:31:17.365567 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:31:17.365310    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-7sh4p" event={"ID":"3fe52df4-b3cc-4ad6-bcf0-5938fb053133","Type":"ContainerDied","Data":"a8427e2bdb321b01563c649a7dff2c683fb1fe3f5333541c693be191f86ff9f4"}
Aug 04 18:31:17.365567 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:31:17.365374    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-7sh4p" event={"ID":"3fe52df4-b3cc-4ad6-bcf0-5938fb053133","Type":"ContainerDied","Data":"baf9a688b492f3130352a85779f532a9517d42e3a1e73670b27b452e352be01b"}
Aug 04 18:31:17.365567 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:31:17.365406    2583 scope.go:117] "RemoveContainer" containerID="a8427e2bdb321b01563c649a7dff2c683fb1fe3f5333541c693be191f86ff9f4"
Aug 04 18:31:17.374983 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:31:17.374965    2583 scope.go:117] "RemoveContainer" containerID="a8427e2bdb321b01563c649a7dff2c683fb1fe3f5333541c693be191f86ff9f4"
Aug 04 18:31:17.375224 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:31:17.375208    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8427e2bdb321b01563c649a7dff2c683fb1fe3f5333541c693be191f86ff9f4\": container with ID starting with a8427e2bdb321b01563c649a7dff2c683fb1fe3f5333541c693be191f86ff9f4 not found: ID does not exist" containerID="a8427e2bdb321b01563c649a7dff2c683fb1fe3f5333541c693be191f86ff9f4"
Aug 04 18:31:17.375263 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:31:17.375233    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8427e2bdb321b01563c649a7dff2c683fb1fe3f5333541c693be191f86ff9f4"} err="failed to get container status \"a8427e2bdb321b01563c649a7dff2c683fb1fe3f5333541c693be191f86ff9f4\": rpc error: code = NotFound desc = could not find container \"a8427e2bdb321b01563c649a7dff2c683fb1fe3f5333541c693be191f86ff9f4\": container with ID starting with a8427e2bdb321b01563c649a7dff2c683fb1fe3f5333541c693be191f86ff9f4 not found: ID does not exist"
Aug 04 18:31:17.391554 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:31:17.391534    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-7sh4p"]
Aug 04 18:31:17.397597 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:31:17.397578    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-7sh4p"]
Aug 04 18:31:18.941940 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:31:18.941907    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3fe52df4-b3cc-4ad6-bcf0-5938fb053133" path="/var/lib/kubelet/pods/3fe52df4-b3cc-4ad6-bcf0-5938fb053133/volumes"
Aug 04 18:31:49.486667 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:31:49.486625    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-899fffbd7-8t97q"]
Aug 04 18:31:49.487641 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:31:49.487610    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3fe52df4-b3cc-4ad6-bcf0-5938fb053133" containerName="main"
Aug 04 18:31:49.487641 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:31:49.487639    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="3fe52df4-b3cc-4ad6-bcf0-5938fb053133" containerName="main"
Aug 04 18:31:49.487826 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:31:49.487718    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="3fe52df4-b3cc-4ad6-bcf0-5938fb053133" containerName="main"
Aug 04 18:31:49.492228 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:31:49.492209    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-899fffbd7-8t97q"
Aug 04 18:31:49.505193 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:31:49.505168    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-899fffbd7-8t97q"]
Aug 04 18:31:49.557826 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:31:49.557791    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7f355dae-56fa-48cb-a01b-5db54f947f41-service-ca\") pod \"console-899fffbd7-8t97q\" (UID: \"7f355dae-56fa-48cb-a01b-5db54f947f41\") " pod="openshift-console/console-899fffbd7-8t97q"
Aug 04 18:31:49.557976 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:31:49.557835    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7f355dae-56fa-48cb-a01b-5db54f947f41-oauth-serving-cert\") pod \"console-899fffbd7-8t97q\" (UID: \"7f355dae-56fa-48cb-a01b-5db54f947f41\") " pod="openshift-console/console-899fffbd7-8t97q"
Aug 04 18:31:49.557976 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:31:49.557899    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7f355dae-56fa-48cb-a01b-5db54f947f41-console-serving-cert\") pod \"console-899fffbd7-8t97q\" (UID: \"7f355dae-56fa-48cb-a01b-5db54f947f41\") " pod="openshift-console/console-899fffbd7-8t97q"
Aug 04 18:31:49.557976 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:31:49.557933    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7f355dae-56fa-48cb-a01b-5db54f947f41-trusted-ca-bundle\") pod \"console-899fffbd7-8t97q\" (UID: \"7f355dae-56fa-48cb-a01b-5db54f947f41\") " pod="openshift-console/console-899fffbd7-8t97q"
Aug 04 18:31:49.557976 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:31:49.557961    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7f355dae-56fa-48cb-a01b-5db54f947f41-console-config\") pod \"console-899fffbd7-8t97q\" (UID: \"7f355dae-56fa-48cb-a01b-5db54f947f41\") " pod="openshift-console/console-899fffbd7-8t97q"
Aug 04 18:31:49.558149 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:31:49.557987    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7f355dae-56fa-48cb-a01b-5db54f947f41-console-oauth-config\") pod \"console-899fffbd7-8t97q\" (UID: \"7f355dae-56fa-48cb-a01b-5db54f947f41\") " pod="openshift-console/console-899fffbd7-8t97q"
Aug 04 18:31:49.558149 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:31:49.558009    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fclqg\" (UniqueName: \"kubernetes.io/projected/7f355dae-56fa-48cb-a01b-5db54f947f41-kube-api-access-fclqg\") pod \"console-899fffbd7-8t97q\" (UID: \"7f355dae-56fa-48cb-a01b-5db54f947f41\") " pod="openshift-console/console-899fffbd7-8t97q"
Aug 04 18:31:49.659236 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:31:49.659205    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fclqg\" (UniqueName: \"kubernetes.io/projected/7f355dae-56fa-48cb-a01b-5db54f947f41-kube-api-access-fclqg\") pod \"console-899fffbd7-8t97q\" (UID: \"7f355dae-56fa-48cb-a01b-5db54f947f41\") " pod="openshift-console/console-899fffbd7-8t97q"
Aug 04 18:31:49.659375 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:31:49.659281    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7f355dae-56fa-48cb-a01b-5db54f947f41-service-ca\") pod \"console-899fffbd7-8t97q\" (UID: \"7f355dae-56fa-48cb-a01b-5db54f947f41\") " pod="openshift-console/console-899fffbd7-8t97q"
Aug 04 18:31:49.659375 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:31:49.659312    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7f355dae-56fa-48cb-a01b-5db54f947f41-oauth-serving-cert\") pod \"console-899fffbd7-8t97q\" (UID: \"7f355dae-56fa-48cb-a01b-5db54f947f41\") " pod="openshift-console/console-899fffbd7-8t97q"
Aug 04 18:31:49.659375 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:31:49.659350    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7f355dae-56fa-48cb-a01b-5db54f947f41-console-serving-cert\") pod \"console-899fffbd7-8t97q\" (UID: \"7f355dae-56fa-48cb-a01b-5db54f947f41\") " pod="openshift-console/console-899fffbd7-8t97q"
Aug 04 18:31:49.659533 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:31:49.659377    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7f355dae-56fa-48cb-a01b-5db54f947f41-trusted-ca-bundle\") pod \"console-899fffbd7-8t97q\" (UID: \"7f355dae-56fa-48cb-a01b-5db54f947f41\") " pod="openshift-console/console-899fffbd7-8t97q"
Aug 04 18:31:49.659533 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:31:49.659404    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7f355dae-56fa-48cb-a01b-5db54f947f41-console-config\") pod \"console-899fffbd7-8t97q\" (UID: \"7f355dae-56fa-48cb-a01b-5db54f947f41\") " pod="openshift-console/console-899fffbd7-8t97q"
Aug 04 18:31:49.659533 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:31:49.659439    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7f355dae-56fa-48cb-a01b-5db54f947f41-console-oauth-config\") pod \"console-899fffbd7-8t97q\" (UID: \"7f355dae-56fa-48cb-a01b-5db54f947f41\") " pod="openshift-console/console-899fffbd7-8t97q"
Aug 04 18:31:49.660166 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:31:49.660133    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7f355dae-56fa-48cb-a01b-5db54f947f41-console-config\") pod \"console-899fffbd7-8t97q\" (UID: \"7f355dae-56fa-48cb-a01b-5db54f947f41\") " pod="openshift-console/console-899fffbd7-8t97q"
Aug 04 18:31:49.660299 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:31:49.660182    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7f355dae-56fa-48cb-a01b-5db54f947f41-oauth-serving-cert\") pod \"console-899fffbd7-8t97q\" (UID: \"7f355dae-56fa-48cb-a01b-5db54f947f41\") " pod="openshift-console/console-899fffbd7-8t97q"
Aug 04 18:31:49.660368 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:31:49.660346    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7f355dae-56fa-48cb-a01b-5db54f947f41-trusted-ca-bundle\") pod \"console-899fffbd7-8t97q\" (UID: \"7f355dae-56fa-48cb-a01b-5db54f947f41\") " pod="openshift-console/console-899fffbd7-8t97q"
Aug 04 18:31:49.660602 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:31:49.660582    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7f355dae-56fa-48cb-a01b-5db54f947f41-service-ca\") pod \"console-899fffbd7-8t97q\" (UID: \"7f355dae-56fa-48cb-a01b-5db54f947f41\") " pod="openshift-console/console-899fffbd7-8t97q"
Aug 04 18:31:49.661821 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:31:49.661802    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7f355dae-56fa-48cb-a01b-5db54f947f41-console-oauth-config\") pod \"console-899fffbd7-8t97q\" (UID: \"7f355dae-56fa-48cb-a01b-5db54f947f41\") " pod="openshift-console/console-899fffbd7-8t97q"
Aug 04 18:31:49.661928 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:31:49.661904    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7f355dae-56fa-48cb-a01b-5db54f947f41-console-serving-cert\") pod \"console-899fffbd7-8t97q\" (UID: \"7f355dae-56fa-48cb-a01b-5db54f947f41\") " pod="openshift-console/console-899fffbd7-8t97q"
Aug 04 18:31:49.667814 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:31:49.667790    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fclqg\" (UniqueName: \"kubernetes.io/projected/7f355dae-56fa-48cb-a01b-5db54f947f41-kube-api-access-fclqg\") pod \"console-899fffbd7-8t97q\" (UID: \"7f355dae-56fa-48cb-a01b-5db54f947f41\") " pod="openshift-console/console-899fffbd7-8t97q"
Aug 04 18:31:49.802973 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:31:49.802904    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-899fffbd7-8t97q"
Aug 04 18:31:49.941448 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:31:49.941421    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-899fffbd7-8t97q"]
Aug 04 18:31:49.943462 ip-10-0-135-124 kubenswrapper[2583]: W0804 18:31:49.943430    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f355dae_56fa_48cb_a01b_5db54f947f41.slice/crio-1210eaa496e9c5e390568e02f7fbf90ec4f41c56d89c0b54a04d29000d92a809 WatchSource:0}: Error finding container 1210eaa496e9c5e390568e02f7fbf90ec4f41c56d89c0b54a04d29000d92a809: Status 404 returned error can't find the container with id 1210eaa496e9c5e390568e02f7fbf90ec4f41c56d89c0b54a04d29000d92a809
Aug 04 18:31:50.542050 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:31:50.542017    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-899fffbd7-8t97q" event={"ID":"7f355dae-56fa-48cb-a01b-5db54f947f41","Type":"ContainerStarted","Data":"ec08b50b6b4e36465abe4de83e967d4f456dc36751a10c5c9a35c7e375024437"}
Aug 04 18:31:50.542439 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:31:50.542067    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-899fffbd7-8t97q" event={"ID":"7f355dae-56fa-48cb-a01b-5db54f947f41","Type":"ContainerStarted","Data":"1210eaa496e9c5e390568e02f7fbf90ec4f41c56d89c0b54a04d29000d92a809"}
Aug 04 18:31:50.564605 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:31:50.564558    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-899fffbd7-8t97q" podStartSLOduration=1.564536352 podStartE2EDuration="1.564536352s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:31:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:31:50.561885656 +0000 UTC m=+1924.143104234" watchObservedRunningTime="2026-08-04 18:31:50.564536352 +0000 UTC m=+1924.145754929"
Aug 04 18:31:59.803055 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:31:59.803009    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-899fffbd7-8t97q"
Aug 04 18:31:59.803571 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:31:59.803182    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-899fffbd7-8t97q"
Aug 04 18:31:59.808133 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:31:59.808110    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-899fffbd7-8t97q"
Aug 04 18:32:00.601238 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:32:00.601207    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-899fffbd7-8t97q"
Aug 04 18:32:00.650816 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:32:00.650778    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-64d47669bd-nr9rw"]
Aug 04 18:32:21.680123 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:32:21.680089    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schej2tnj"]
Aug 04 18:32:21.688822 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:32:21.688789    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schej2tnj"
Aug 04 18:32:21.693832 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:32:21.693807    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"kube-root-ca.crt\""
Aug 04 18:32:21.693961 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:32:21.693849    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisvb11a9c9832b99b016bc8f8e0ea095712-kserve-self-signed-certs\""
Aug 04 18:32:21.693961 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:32:21.693851    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"default-dockercfg-g5ksp\""
Aug 04 18:32:21.693961 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:32:21.693892    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"openshift-service-ca.crt\""
Aug 04 18:32:21.693961 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:32:21.693901    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-epp-sa-dockercfg-flgx4\""
Aug 04 18:32:21.701546 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:32:21.701525    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schej2tnj"]
Aug 04 18:32:21.838403 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:32:21.838370    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b8fcf12-6c17-4f06-97c5-a54d458d79b4-tls-certs\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schej2tnj\" (UID: \"4b8fcf12-6c17-4f06-97c5-a54d458d79b4\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schej2tnj"
Aug 04 18:32:21.838588 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:32:21.838432    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ds9dm\" (UniqueName: \"kubernetes.io/projected/4b8fcf12-6c17-4f06-97c5-a54d458d79b4-kube-api-access-ds9dm\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schej2tnj\" (UID: \"4b8fcf12-6c17-4f06-97c5-a54d458d79b4\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schej2tnj"
Aug 04 18:32:21.939260 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:32:21.939222    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ds9dm\" (UniqueName: \"kubernetes.io/projected/4b8fcf12-6c17-4f06-97c5-a54d458d79b4-kube-api-access-ds9dm\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schej2tnj\" (UID: \"4b8fcf12-6c17-4f06-97c5-a54d458d79b4\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schej2tnj"
Aug 04 18:32:21.939450 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:32:21.939308    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b8fcf12-6c17-4f06-97c5-a54d458d79b4-tls-certs\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schej2tnj\" (UID: \"4b8fcf12-6c17-4f06-97c5-a54d458d79b4\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schej2tnj"
Aug 04 18:32:21.941865 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:32:21.941839    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b8fcf12-6c17-4f06-97c5-a54d458d79b4-tls-certs\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schej2tnj\" (UID: \"4b8fcf12-6c17-4f06-97c5-a54d458d79b4\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schej2tnj"
Aug 04 18:32:21.950106 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:32:21.950079    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ds9dm\" (UniqueName: \"kubernetes.io/projected/4b8fcf12-6c17-4f06-97c5-a54d458d79b4-kube-api-access-ds9dm\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schej2tnj\" (UID: \"4b8fcf12-6c17-4f06-97c5-a54d458d79b4\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schej2tnj"
Aug 04 18:32:22.000768 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:32:22.000737    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schej2tnj"
Aug 04 18:32:22.342046 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:32:22.341961    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schej2tnj"]
Aug 04 18:32:22.345324 ip-10-0-135-124 kubenswrapper[2583]: W0804 18:32:22.345287    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b8fcf12_6c17_4f06_97c5_a54d458d79b4.slice/crio-2789cbe37371fe0005d09aa53c3da6993bee197d387737c96b4344cc3976b7eb WatchSource:0}: Error finding container 2789cbe37371fe0005d09aa53c3da6993bee197d387737c96b4344cc3976b7eb: Status 404 returned error can't find the container with id 2789cbe37371fe0005d09aa53c3da6993bee197d387737c96b4344cc3976b7eb
Aug 04 18:32:22.709985 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:32:22.709951    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schej2tnj" event={"ID":"4b8fcf12-6c17-4f06-97c5-a54d458d79b4","Type":"ContainerStarted","Data":"1b35b3a9480cb5495e1b3c3002569d52be43f5deb1e0f3eb336cf5337f89ba86"}
Aug 04 18:32:22.710368 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:32:22.710014    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schej2tnj"
Aug 04 18:32:22.710368 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:32:22.710032    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schej2tnj" event={"ID":"4b8fcf12-6c17-4f06-97c5-a54d458d79b4","Type":"ContainerStarted","Data":"2789cbe37371fe0005d09aa53c3da6993bee197d387737c96b4344cc3976b7eb"}
Aug 04 18:32:22.731296 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:32:22.731247    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schej2tnj" podStartSLOduration=1.731233117 podStartE2EDuration="1.731233117s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:32:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:32:22.729000643 +0000 UTC m=+1956.310219221" watchObservedRunningTime="2026-08-04 18:32:22.731233117 +0000 UTC m=+1956.312451693"
Aug 04 18:32:25.675056 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:32:25.675017    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-64d47669bd-nr9rw" podUID="5c2ebfd2-7cbb-468c-97ee-2867e9cc4b51" containerName="console" containerID="cri-o://1c60923780a402052b0efd4fe0c8a674d906a8041766e29a302f84b2e9b980ce" gracePeriod=15
Aug 04 18:32:25.937004 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:32:25.936976    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-64d47669bd-nr9rw"
Aug 04 18:32:26.074837 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:32:26.074803    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5c2ebfd2-7cbb-468c-97ee-2867e9cc4b51-service-ca\") pod \"5c2ebfd2-7cbb-468c-97ee-2867e9cc4b51\" (UID: \"5c2ebfd2-7cbb-468c-97ee-2867e9cc4b51\") "
Aug 04 18:32:26.074837 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:32:26.074837    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5c2ebfd2-7cbb-468c-97ee-2867e9cc4b51-trusted-ca-bundle\") pod \"5c2ebfd2-7cbb-468c-97ee-2867e9cc4b51\" (UID: \"5c2ebfd2-7cbb-468c-97ee-2867e9cc4b51\") "
Aug 04 18:32:26.075038 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:32:26.074860    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5c2ebfd2-7cbb-468c-97ee-2867e9cc4b51-console-serving-cert\") pod \"5c2ebfd2-7cbb-468c-97ee-2867e9cc4b51\" (UID: \"5c2ebfd2-7cbb-468c-97ee-2867e9cc4b51\") "
Aug 04 18:32:26.075038 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:32:26.074897    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m5fs6\" (UniqueName: \"kubernetes.io/projected/5c2ebfd2-7cbb-468c-97ee-2867e9cc4b51-kube-api-access-m5fs6\") pod \"5c2ebfd2-7cbb-468c-97ee-2867e9cc4b51\" (UID: \"5c2ebfd2-7cbb-468c-97ee-2867e9cc4b51\") "
Aug 04 18:32:26.075038 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:32:26.074943    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5c2ebfd2-7cbb-468c-97ee-2867e9cc4b51-oauth-serving-cert\") pod \"5c2ebfd2-7cbb-468c-97ee-2867e9cc4b51\" (UID: \"5c2ebfd2-7cbb-468c-97ee-2867e9cc4b51\") "
Aug 04 18:32:26.075038 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:32:26.074965    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5c2ebfd2-7cbb-468c-97ee-2867e9cc4b51-console-oauth-config\") pod \"5c2ebfd2-7cbb-468c-97ee-2867e9cc4b51\" (UID: \"5c2ebfd2-7cbb-468c-97ee-2867e9cc4b51\") "
Aug 04 18:32:26.075241 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:32:26.075127    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5c2ebfd2-7cbb-468c-97ee-2867e9cc4b51-console-config\") pod \"5c2ebfd2-7cbb-468c-97ee-2867e9cc4b51\" (UID: \"5c2ebfd2-7cbb-468c-97ee-2867e9cc4b51\") "
Aug 04 18:32:26.075402 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:32:26.075375    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c2ebfd2-7cbb-468c-97ee-2867e9cc4b51-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "5c2ebfd2-7cbb-468c-97ee-2867e9cc4b51" (UID: "5c2ebfd2-7cbb-468c-97ee-2867e9cc4b51"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:32:26.075599 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:32:26.075545    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c2ebfd2-7cbb-468c-97ee-2867e9cc4b51-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "5c2ebfd2-7cbb-468c-97ee-2867e9cc4b51" (UID: "5c2ebfd2-7cbb-468c-97ee-2867e9cc4b51"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:32:26.075675 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:32:26.075639    2583 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5c2ebfd2-7cbb-468c-97ee-2867e9cc4b51-trusted-ca-bundle\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:32:26.075675 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:32:26.075659    2583 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5c2ebfd2-7cbb-468c-97ee-2867e9cc4b51-oauth-serving-cert\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:32:26.075845 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:32:26.075818    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c2ebfd2-7cbb-468c-97ee-2867e9cc4b51-service-ca" (OuterVolumeSpecName: "service-ca") pod "5c2ebfd2-7cbb-468c-97ee-2867e9cc4b51" (UID: "5c2ebfd2-7cbb-468c-97ee-2867e9cc4b51"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:32:26.075950 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:32:26.075906    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c2ebfd2-7cbb-468c-97ee-2867e9cc4b51-console-config" (OuterVolumeSpecName: "console-config") pod "5c2ebfd2-7cbb-468c-97ee-2867e9cc4b51" (UID: "5c2ebfd2-7cbb-468c-97ee-2867e9cc4b51"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:32:26.077084 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:32:26.077059    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c2ebfd2-7cbb-468c-97ee-2867e9cc4b51-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "5c2ebfd2-7cbb-468c-97ee-2867e9cc4b51" (UID: "5c2ebfd2-7cbb-468c-97ee-2867e9cc4b51"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:32:26.077222 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:32:26.077204    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c2ebfd2-7cbb-468c-97ee-2867e9cc4b51-kube-api-access-m5fs6" (OuterVolumeSpecName: "kube-api-access-m5fs6") pod "5c2ebfd2-7cbb-468c-97ee-2867e9cc4b51" (UID: "5c2ebfd2-7cbb-468c-97ee-2867e9cc4b51"). InnerVolumeSpecName "kube-api-access-m5fs6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:32:26.077310 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:32:26.077296    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c2ebfd2-7cbb-468c-97ee-2867e9cc4b51-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "5c2ebfd2-7cbb-468c-97ee-2867e9cc4b51" (UID: "5c2ebfd2-7cbb-468c-97ee-2867e9cc4b51"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:32:26.176349 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:32:26.176323    2583 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5c2ebfd2-7cbb-468c-97ee-2867e9cc4b51-console-config\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:32:26.176349 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:32:26.176345    2583 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5c2ebfd2-7cbb-468c-97ee-2867e9cc4b51-service-ca\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:32:26.176349 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:32:26.176354    2583 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5c2ebfd2-7cbb-468c-97ee-2867e9cc4b51-console-serving-cert\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:32:26.176576 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:32:26.176364    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m5fs6\" (UniqueName: \"kubernetes.io/projected/5c2ebfd2-7cbb-468c-97ee-2867e9cc4b51-kube-api-access-m5fs6\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:32:26.176576 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:32:26.176373    2583 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5c2ebfd2-7cbb-468c-97ee-2867e9cc4b51-console-oauth-config\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:32:26.740225 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:32:26.740191    2583 generic.go:356] "Generic (PLEG): container finished" podID="5c2ebfd2-7cbb-468c-97ee-2867e9cc4b51" containerID="1c60923780a402052b0efd4fe0c8a674d906a8041766e29a302f84b2e9b980ce" exitCode=0
Aug 04 18:32:26.740755 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:32:26.740233    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-64d47669bd-nr9rw"
Aug 04 18:32:26.740755 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:32:26.740277    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-64d47669bd-nr9rw" event={"ID":"5c2ebfd2-7cbb-468c-97ee-2867e9cc4b51","Type":"ContainerDied","Data":"1c60923780a402052b0efd4fe0c8a674d906a8041766e29a302f84b2e9b980ce"}
Aug 04 18:32:26.740755 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:32:26.740329    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-64d47669bd-nr9rw" event={"ID":"5c2ebfd2-7cbb-468c-97ee-2867e9cc4b51","Type":"ContainerDied","Data":"2b9b311de79b1d68ebf24d947a1ac60ca72713868f8751c026196b3f5463c5b1"}
Aug 04 18:32:26.740755 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:32:26.740354    2583 scope.go:117] "RemoveContainer" containerID="1c60923780a402052b0efd4fe0c8a674d906a8041766e29a302f84b2e9b980ce"
Aug 04 18:32:26.750759 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:32:26.750738    2583 scope.go:117] "RemoveContainer" containerID="1c60923780a402052b0efd4fe0c8a674d906a8041766e29a302f84b2e9b980ce"
Aug 04 18:32:26.751059 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:32:26.751039    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c60923780a402052b0efd4fe0c8a674d906a8041766e29a302f84b2e9b980ce\": container with ID starting with 1c60923780a402052b0efd4fe0c8a674d906a8041766e29a302f84b2e9b980ce not found: ID does not exist" containerID="1c60923780a402052b0efd4fe0c8a674d906a8041766e29a302f84b2e9b980ce"
Aug 04 18:32:26.751125 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:32:26.751071    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c60923780a402052b0efd4fe0c8a674d906a8041766e29a302f84b2e9b980ce"} err="failed to get container status \"1c60923780a402052b0efd4fe0c8a674d906a8041766e29a302f84b2e9b980ce\": rpc error: code = NotFound desc = could not find container \"1c60923780a402052b0efd4fe0c8a674d906a8041766e29a302f84b2e9b980ce\": container with ID starting with 1c60923780a402052b0efd4fe0c8a674d906a8041766e29a302f84b2e9b980ce not found: ID does not exist"
Aug 04 18:32:26.766429 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:32:26.766403    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-64d47669bd-nr9rw"]
Aug 04 18:32:26.770722 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:32:26.770698    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-64d47669bd-nr9rw"]
Aug 04 18:32:26.942409 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:32:26.942379    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5c2ebfd2-7cbb-468c-97ee-2867e9cc4b51" path="/var/lib/kubelet/pods/5c2ebfd2-7cbb-468c-97ee-2867e9cc4b51/volumes"
Aug 04 18:32:49.268533 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:32:49.268497    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-rtmbx"]
Aug 04 18:32:53.717158 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:32:53.717130    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schej2tnj"
Aug 04 18:33:04.289764 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:33:04.289697    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-rtmbx" podUID="4f5add6e-57c5-4940-b289-67ef3c98449c" containerName="main" containerID="cri-o://a840d83698b4038a35a38d99ccca94757ced22af2a81c1ad086cec4a33e25e94" gracePeriod=45
Aug 04 18:33:04.359614 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:33:04.359578    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-rtmbx" podUID="4f5add6e-57c5-4940-b289-67ef3c98449c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": dial tcp 10.132.0.67:8000: connect: connection refused"
Aug 04 18:33:04.540016 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:33:04.539953    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-rtmbx"
Aug 04 18:33:04.617809 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:33:04.617778    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f5add6e-57c5-4940-b289-67ef3c98449c-home\") pod \"4f5add6e-57c5-4940-b289-67ef3c98449c\" (UID: \"4f5add6e-57c5-4940-b289-67ef3c98449c\") "
Aug 04 18:33:04.617995 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:33:04.617832    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f5add6e-57c5-4940-b289-67ef3c98449c-tmp-dir\") pod \"4f5add6e-57c5-4940-b289-67ef3c98449c\" (UID: \"4f5add6e-57c5-4940-b289-67ef3c98449c\") "
Aug 04 18:33:04.617995 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:33:04.617863    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f5add6e-57c5-4940-b289-67ef3c98449c-tls-certs\") pod \"4f5add6e-57c5-4940-b289-67ef3c98449c\" (UID: \"4f5add6e-57c5-4940-b289-67ef3c98449c\") "
Aug 04 18:33:04.617995 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:33:04.617902    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f5add6e-57c5-4940-b289-67ef3c98449c-model-cache\") pod \"4f5add6e-57c5-4940-b289-67ef3c98449c\" (UID: \"4f5add6e-57c5-4940-b289-67ef3c98449c\") "
Aug 04 18:33:04.617995 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:33:04.617955    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2fgxj\" (UniqueName: \"kubernetes.io/projected/4f5add6e-57c5-4940-b289-67ef3c98449c-kube-api-access-2fgxj\") pod \"4f5add6e-57c5-4940-b289-67ef3c98449c\" (UID: \"4f5add6e-57c5-4940-b289-67ef3c98449c\") "
Aug 04 18:33:04.618227 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:33:04.618096    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f5add6e-57c5-4940-b289-67ef3c98449c-home" (OuterVolumeSpecName: "home") pod "4f5add6e-57c5-4940-b289-67ef3c98449c" (UID: "4f5add6e-57c5-4940-b289-67ef3c98449c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:33:04.618227 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:33:04.618122    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f5add6e-57c5-4940-b289-67ef3c98449c-dshm\") pod \"4f5add6e-57c5-4940-b289-67ef3c98449c\" (UID: \"4f5add6e-57c5-4940-b289-67ef3c98449c\") "
Aug 04 18:33:04.618227 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:33:04.618140    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f5add6e-57c5-4940-b289-67ef3c98449c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4f5add6e-57c5-4940-b289-67ef3c98449c" (UID: "4f5add6e-57c5-4940-b289-67ef3c98449c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:33:04.618377 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:33:04.618237    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f5add6e-57c5-4940-b289-67ef3c98449c-model-cache" (OuterVolumeSpecName: "model-cache") pod "4f5add6e-57c5-4940-b289-67ef3c98449c" (UID: "4f5add6e-57c5-4940-b289-67ef3c98449c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:33:04.618458 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:33:04.618441    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f5add6e-57c5-4940-b289-67ef3c98449c-model-cache\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:33:04.618539 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:33:04.618464    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f5add6e-57c5-4940-b289-67ef3c98449c-home\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:33:04.618539 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:33:04.618504    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f5add6e-57c5-4940-b289-67ef3c98449c-tmp-dir\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:33:04.620276 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:33:04.620252    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f5add6e-57c5-4940-b289-67ef3c98449c-dshm" (OuterVolumeSpecName: "dshm") pod "4f5add6e-57c5-4940-b289-67ef3c98449c" (UID: "4f5add6e-57c5-4940-b289-67ef3c98449c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:33:04.620347 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:33:04.620297    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f5add6e-57c5-4940-b289-67ef3c98449c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4f5add6e-57c5-4940-b289-67ef3c98449c" (UID: "4f5add6e-57c5-4940-b289-67ef3c98449c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:33:04.620401 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:33:04.620382    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f5add6e-57c5-4940-b289-67ef3c98449c-kube-api-access-2fgxj" (OuterVolumeSpecName: "kube-api-access-2fgxj") pod "4f5add6e-57c5-4940-b289-67ef3c98449c" (UID: "4f5add6e-57c5-4940-b289-67ef3c98449c"). InnerVolumeSpecName "kube-api-access-2fgxj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:33:04.718961 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:33:04.718925    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f5add6e-57c5-4940-b289-67ef3c98449c-tls-certs\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:33:04.718961 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:33:04.718958    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2fgxj\" (UniqueName: \"kubernetes.io/projected/4f5add6e-57c5-4940-b289-67ef3c98449c-kube-api-access-2fgxj\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:33:04.718961 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:33:04.718968    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f5add6e-57c5-4940-b289-67ef3c98449c-dshm\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:33:04.945730 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:33:04.945693    2583 generic.go:356] "Generic (PLEG): container finished" podID="4f5add6e-57c5-4940-b289-67ef3c98449c" containerID="a840d83698b4038a35a38d99ccca94757ced22af2a81c1ad086cec4a33e25e94" exitCode=0
Aug 04 18:33:04.945935 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:33:04.945741    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-rtmbx"
Aug 04 18:33:04.945935 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:33:04.945776    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-rtmbx" event={"ID":"4f5add6e-57c5-4940-b289-67ef3c98449c","Type":"ContainerDied","Data":"a840d83698b4038a35a38d99ccca94757ced22af2a81c1ad086cec4a33e25e94"}
Aug 04 18:33:04.945935 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:33:04.945834    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-rtmbx" event={"ID":"4f5add6e-57c5-4940-b289-67ef3c98449c","Type":"ContainerDied","Data":"63c9bea715ac341e9a3c8a786b9cf90f34d0035890197cbd4cd72c03f8df41ca"}
Aug 04 18:33:04.945935 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:33:04.945857    2583 scope.go:117] "RemoveContainer" containerID="a840d83698b4038a35a38d99ccca94757ced22af2a81c1ad086cec4a33e25e94"
Aug 04 18:33:04.955978 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:33:04.955956    2583 scope.go:117] "RemoveContainer" containerID="a840d83698b4038a35a38d99ccca94757ced22af2a81c1ad086cec4a33e25e94"
Aug 04 18:33:04.956245 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:33:04.956225    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a840d83698b4038a35a38d99ccca94757ced22af2a81c1ad086cec4a33e25e94\": container with ID starting with a840d83698b4038a35a38d99ccca94757ced22af2a81c1ad086cec4a33e25e94 not found: ID does not exist" containerID="a840d83698b4038a35a38d99ccca94757ced22af2a81c1ad086cec4a33e25e94"
Aug 04 18:33:04.956308 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:33:04.956259    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a840d83698b4038a35a38d99ccca94757ced22af2a81c1ad086cec4a33e25e94"} err="failed to get container status \"a840d83698b4038a35a38d99ccca94757ced22af2a81c1ad086cec4a33e25e94\": rpc error: code = NotFound desc = could not find container \"a840d83698b4038a35a38d99ccca94757ced22af2a81c1ad086cec4a33e25e94\": container with ID starting with a840d83698b4038a35a38d99ccca94757ced22af2a81c1ad086cec4a33e25e94 not found: ID does not exist"
Aug 04 18:33:04.970376 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:33:04.970344    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-rtmbx"]
Aug 04 18:33:04.975844 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:33:04.975819    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v1-kserve-6cb797fb75-rtmbx"]
Aug 04 18:33:06.943672 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:33:06.943637    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4f5add6e-57c5-4940-b289-67ef3c98449c" path="/var/lib/kubelet/pods/4f5add6e-57c5-4940-b289-67ef3c98449c/volumes"
Aug 04 18:33:08.222955 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:33:08.222908    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-n92sg"]
Aug 04 18:33:08.223490 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:33:08.223446    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c2ebfd2-7cbb-468c-97ee-2867e9cc4b51" containerName="console"
Aug 04 18:33:08.223490 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:33:08.223464    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c2ebfd2-7cbb-468c-97ee-2867e9cc4b51" containerName="console"
Aug 04 18:33:08.223582 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:33:08.223507    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f5add6e-57c5-4940-b289-67ef3c98449c" containerName="main"
Aug 04 18:33:08.223582 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:33:08.223516    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f5add6e-57c5-4940-b289-67ef3c98449c" containerName="main"
Aug 04 18:33:08.223661 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:33:08.223591    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="4f5add6e-57c5-4940-b289-67ef3c98449c" containerName="main"
Aug 04 18:33:08.223661 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:33:08.223602    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="5c2ebfd2-7cbb-468c-97ee-2867e9cc4b51" containerName="console"
Aug 04 18:33:08.227828 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:33:08.227810    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-n92sg"
Aug 04 18:33:08.233362 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:33:08.233339    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-leave-group-5b69d57a\"/\"leave-v1-kserve-self-signed-certs\""
Aug 04 18:33:08.234580 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:33:08.234301    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-leave-group-5b69d57a\"/\"default-dockercfg-4zcxc\""
Aug 04 18:33:08.234580 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:33:08.234301    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-leave-group-5b69d57a\"/\"openshift-service-ca.crt\""
Aug 04 18:33:08.234580 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:33:08.234339    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-leave-group-5b69d57a\"/\"kube-root-ca.crt\""
Aug 04 18:33:08.240090 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:33:08.240063    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-n92sg"]
Aug 04 18:33:08.352177 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:33:08.352137    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc9aa361-3e43-4847-9573-ac0e1b9dbef7-tmp-dir\") pod \"leave-v1-kserve-65684d4f5-n92sg\" (UID: \"cc9aa361-3e43-4847-9573-ac0e1b9dbef7\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-n92sg"
Aug 04 18:33:08.352356 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:33:08.352203    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrqpb\" (UniqueName: \"kubernetes.io/projected/cc9aa361-3e43-4847-9573-ac0e1b9dbef7-kube-api-access-xrqpb\") pod \"leave-v1-kserve-65684d4f5-n92sg\" (UID: \"cc9aa361-3e43-4847-9573-ac0e1b9dbef7\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-n92sg"
Aug 04 18:33:08.352356 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:33:08.352248    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc9aa361-3e43-4847-9573-ac0e1b9dbef7-tls-certs\") pod \"leave-v1-kserve-65684d4f5-n92sg\" (UID: \"cc9aa361-3e43-4847-9573-ac0e1b9dbef7\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-n92sg"
Aug 04 18:33:08.352356 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:33:08.352284    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc9aa361-3e43-4847-9573-ac0e1b9dbef7-home\") pod \"leave-v1-kserve-65684d4f5-n92sg\" (UID: \"cc9aa361-3e43-4847-9573-ac0e1b9dbef7\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-n92sg"
Aug 04 18:33:08.352356 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:33:08.352301    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc9aa361-3e43-4847-9573-ac0e1b9dbef7-model-cache\") pod \"leave-v1-kserve-65684d4f5-n92sg\" (UID: \"cc9aa361-3e43-4847-9573-ac0e1b9dbef7\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-n92sg"
Aug 04 18:33:08.352356 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:33:08.352319    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc9aa361-3e43-4847-9573-ac0e1b9dbef7-dshm\") pod \"leave-v1-kserve-65684d4f5-n92sg\" (UID: \"cc9aa361-3e43-4847-9573-ac0e1b9dbef7\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-n92sg"
Aug 04 18:33:08.453757 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:33:08.453713    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrqpb\" (UniqueName: \"kubernetes.io/projected/cc9aa361-3e43-4847-9573-ac0e1b9dbef7-kube-api-access-xrqpb\") pod \"leave-v1-kserve-65684d4f5-n92sg\" (UID: \"cc9aa361-3e43-4847-9573-ac0e1b9dbef7\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-n92sg"
Aug 04 18:33:08.453972 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:33:08.453766    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc9aa361-3e43-4847-9573-ac0e1b9dbef7-tls-certs\") pod \"leave-v1-kserve-65684d4f5-n92sg\" (UID: \"cc9aa361-3e43-4847-9573-ac0e1b9dbef7\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-n92sg"
Aug 04 18:33:08.453972 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:33:08.453810    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc9aa361-3e43-4847-9573-ac0e1b9dbef7-home\") pod \"leave-v1-kserve-65684d4f5-n92sg\" (UID: \"cc9aa361-3e43-4847-9573-ac0e1b9dbef7\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-n92sg"
Aug 04 18:33:08.454100 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:33:08.453964    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc9aa361-3e43-4847-9573-ac0e1b9dbef7-model-cache\") pod \"leave-v1-kserve-65684d4f5-n92sg\" (UID: \"cc9aa361-3e43-4847-9573-ac0e1b9dbef7\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-n92sg"
Aug 04 18:33:08.454100 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:33:08.454014    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc9aa361-3e43-4847-9573-ac0e1b9dbef7-dshm\") pod \"leave-v1-kserve-65684d4f5-n92sg\" (UID: \"cc9aa361-3e43-4847-9573-ac0e1b9dbef7\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-n92sg"
Aug 04 18:33:08.454213 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:33:08.454115    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc9aa361-3e43-4847-9573-ac0e1b9dbef7-tmp-dir\") pod \"leave-v1-kserve-65684d4f5-n92sg\" (UID: \"cc9aa361-3e43-4847-9573-ac0e1b9dbef7\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-n92sg"
Aug 04 18:33:08.454270 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:33:08.454243    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc9aa361-3e43-4847-9573-ac0e1b9dbef7-home\") pod \"leave-v1-kserve-65684d4f5-n92sg\" (UID: \"cc9aa361-3e43-4847-9573-ac0e1b9dbef7\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-n92sg"
Aug 04 18:33:08.454326 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:33:08.454304    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc9aa361-3e43-4847-9573-ac0e1b9dbef7-model-cache\") pod \"leave-v1-kserve-65684d4f5-n92sg\" (UID: \"cc9aa361-3e43-4847-9573-ac0e1b9dbef7\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-n92sg"
Aug 04 18:33:08.454436 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:33:08.454421    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc9aa361-3e43-4847-9573-ac0e1b9dbef7-tmp-dir\") pod \"leave-v1-kserve-65684d4f5-n92sg\" (UID: \"cc9aa361-3e43-4847-9573-ac0e1b9dbef7\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-n92sg"
Aug 04 18:33:08.456207 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:33:08.456183    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc9aa361-3e43-4847-9573-ac0e1b9dbef7-dshm\") pod \"leave-v1-kserve-65684d4f5-n92sg\" (UID: \"cc9aa361-3e43-4847-9573-ac0e1b9dbef7\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-n92sg"
Aug 04 18:33:08.456672 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:33:08.456649    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc9aa361-3e43-4847-9573-ac0e1b9dbef7-tls-certs\") pod \"leave-v1-kserve-65684d4f5-n92sg\" (UID: \"cc9aa361-3e43-4847-9573-ac0e1b9dbef7\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-n92sg"
Aug 04 18:33:08.462620 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:33:08.462590    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrqpb\" (UniqueName: \"kubernetes.io/projected/cc9aa361-3e43-4847-9573-ac0e1b9dbef7-kube-api-access-xrqpb\") pod \"leave-v1-kserve-65684d4f5-n92sg\" (UID: \"cc9aa361-3e43-4847-9573-ac0e1b9dbef7\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-n92sg"
Aug 04 18:33:08.538423 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:33:08.538325    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-n92sg"
Aug 04 18:33:08.678593 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:33:08.678561    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-n92sg"]
Aug 04 18:33:08.970505 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:33:08.970444    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-n92sg" event={"ID":"cc9aa361-3e43-4847-9573-ac0e1b9dbef7","Type":"ContainerStarted","Data":"39febdf09b18757638a7482bfe0f93657dc44d9887a2cdb64d69770b235209d0"}
Aug 04 18:33:08.970689 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:33:08.970523    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-n92sg" event={"ID":"cc9aa361-3e43-4847-9573-ac0e1b9dbef7","Type":"ContainerStarted","Data":"51605489e48f94bc2df90addfb65b889026edfe548e2210fa8a494212c51cfed"}
Aug 04 18:33:08.993785 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:33:08.993729    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-n92sg" podStartSLOduration=0.993712559 podStartE2EDuration="993.712559ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:33:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:33:08.991817154 +0000 UTC m=+2002.573035733" watchObservedRunningTime="2026-08-04 18:33:08.993712559 +0000 UTC m=+2002.574931137"
Aug 04 18:33:09.539085 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:33:09.539048    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-n92sg"
Aug 04 18:33:18.539330 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:33:18.539285    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-n92sg"
Aug 04 18:33:18.553224 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:33:18.553195    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-n92sg"
Aug 04 18:33:19.032109 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:33:19.032080    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-n92sg"
Aug 04 18:33:53.220336 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:33:53.220292    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schej2tnj"]
Aug 04 18:33:53.717532 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:33:53.717463    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schej2tnj" podUID="4b8fcf12-6c17-4f06-97c5-a54d458d79b4" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Aug 04 18:34:00.451285 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:34:00.451241    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-n92sg"]
Aug 04 18:34:03.717633 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:34:03.717588    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schej2tnj" podUID="4b8fcf12-6c17-4f06-97c5-a54d458d79b4" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Aug 04 18:34:08.246970 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:34:08.246911    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schej2tnj" podUID="4b8fcf12-6c17-4f06-97c5-a54d458d79b4" containerName="main" containerID="cri-o://1b35b3a9480cb5495e1b3c3002569d52be43f5deb1e0f3eb336cf5337f89ba86" gracePeriod=45
Aug 04 18:34:08.499938 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:34:08.499864    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schej2tnj"
Aug 04 18:34:08.585377 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:34:08.585337    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b8fcf12-6c17-4f06-97c5-a54d458d79b4-tls-certs\") pod \"4b8fcf12-6c17-4f06-97c5-a54d458d79b4\" (UID: \"4b8fcf12-6c17-4f06-97c5-a54d458d79b4\") "
Aug 04 18:34:08.585614 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:34:08.585396    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ds9dm\" (UniqueName: \"kubernetes.io/projected/4b8fcf12-6c17-4f06-97c5-a54d458d79b4-kube-api-access-ds9dm\") pod \"4b8fcf12-6c17-4f06-97c5-a54d458d79b4\" (UID: \"4b8fcf12-6c17-4f06-97c5-a54d458d79b4\") "
Aug 04 18:34:08.587723 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:34:08.587687    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b8fcf12-6c17-4f06-97c5-a54d458d79b4-kube-api-access-ds9dm" (OuterVolumeSpecName: "kube-api-access-ds9dm") pod "4b8fcf12-6c17-4f06-97c5-a54d458d79b4" (UID: "4b8fcf12-6c17-4f06-97c5-a54d458d79b4"). InnerVolumeSpecName "kube-api-access-ds9dm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:34:08.587836 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:34:08.587753    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b8fcf12-6c17-4f06-97c5-a54d458d79b4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4b8fcf12-6c17-4f06-97c5-a54d458d79b4" (UID: "4b8fcf12-6c17-4f06-97c5-a54d458d79b4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:34:08.686641 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:34:08.686603    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b8fcf12-6c17-4f06-97c5-a54d458d79b4-tls-certs\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:34:08.686641 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:34:08.686633    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ds9dm\" (UniqueName: \"kubernetes.io/projected/4b8fcf12-6c17-4f06-97c5-a54d458d79b4-kube-api-access-ds9dm\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:34:09.299817 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:34:09.299726    2583 generic.go:356] "Generic (PLEG): container finished" podID="4b8fcf12-6c17-4f06-97c5-a54d458d79b4" containerID="1b35b3a9480cb5495e1b3c3002569d52be43f5deb1e0f3eb336cf5337f89ba86" exitCode=0
Aug 04 18:34:09.299817 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:34:09.299772    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schej2tnj"
Aug 04 18:34:09.299817 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:34:09.299786    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schej2tnj" event={"ID":"4b8fcf12-6c17-4f06-97c5-a54d458d79b4","Type":"ContainerDied","Data":"1b35b3a9480cb5495e1b3c3002569d52be43f5deb1e0f3eb336cf5337f89ba86"}
Aug 04 18:34:09.300321 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:34:09.299828    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schej2tnj" event={"ID":"4b8fcf12-6c17-4f06-97c5-a54d458d79b4","Type":"ContainerDied","Data":"2789cbe37371fe0005d09aa53c3da6993bee197d387737c96b4344cc3976b7eb"}
Aug 04 18:34:09.300321 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:34:09.299852    2583 scope.go:117] "RemoveContainer" containerID="1b35b3a9480cb5495e1b3c3002569d52be43f5deb1e0f3eb336cf5337f89ba86"
Aug 04 18:34:09.308932 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:34:09.308910    2583 scope.go:117] "RemoveContainer" containerID="1b35b3a9480cb5495e1b3c3002569d52be43f5deb1e0f3eb336cf5337f89ba86"
Aug 04 18:34:09.309264 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:34:09.309245    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b35b3a9480cb5495e1b3c3002569d52be43f5deb1e0f3eb336cf5337f89ba86\": container with ID starting with 1b35b3a9480cb5495e1b3c3002569d52be43f5deb1e0f3eb336cf5337f89ba86 not found: ID does not exist" containerID="1b35b3a9480cb5495e1b3c3002569d52be43f5deb1e0f3eb336cf5337f89ba86"
Aug 04 18:34:09.309327 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:34:09.309271    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b35b3a9480cb5495e1b3c3002569d52be43f5deb1e0f3eb336cf5337f89ba86"} err="failed to get container status \"1b35b3a9480cb5495e1b3c3002569d52be43f5deb1e0f3eb336cf5337f89ba86\": rpc error: code = NotFound desc = could not find container \"1b35b3a9480cb5495e1b3c3002569d52be43f5deb1e0f3eb336cf5337f89ba86\": container with ID starting with 1b35b3a9480cb5495e1b3c3002569d52be43f5deb1e0f3eb336cf5337f89ba86 not found: ID does not exist"
Aug 04 18:34:09.319669 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:34:09.319633    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schej2tnj"]
Aug 04 18:34:09.321687 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:34:09.321663    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schej2tnj"]
Aug 04 18:34:10.942950 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:34:10.942912    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4b8fcf12-6c17-4f06-97c5-a54d458d79b4" path="/var/lib/kubelet/pods/4b8fcf12-6c17-4f06-97c5-a54d458d79b4/volumes"
Aug 04 18:34:15.471203 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:34:15.471163    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-n92sg" podUID="cc9aa361-3e43-4847-9573-ac0e1b9dbef7" containerName="main" containerID="cri-o://39febdf09b18757638a7482bfe0f93657dc44d9887a2cdb64d69770b235209d0" gracePeriod=45
Aug 04 18:34:15.722225 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:34:15.722159    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-n92sg"
Aug 04 18:34:15.757113 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:34:15.757078    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc9aa361-3e43-4847-9573-ac0e1b9dbef7-tmp-dir\") pod \"cc9aa361-3e43-4847-9573-ac0e1b9dbef7\" (UID: \"cc9aa361-3e43-4847-9573-ac0e1b9dbef7\") "
Aug 04 18:34:15.757305 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:34:15.757136    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xrqpb\" (UniqueName: \"kubernetes.io/projected/cc9aa361-3e43-4847-9573-ac0e1b9dbef7-kube-api-access-xrqpb\") pod \"cc9aa361-3e43-4847-9573-ac0e1b9dbef7\" (UID: \"cc9aa361-3e43-4847-9573-ac0e1b9dbef7\") "
Aug 04 18:34:15.757305 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:34:15.757160    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc9aa361-3e43-4847-9573-ac0e1b9dbef7-home\") pod \"cc9aa361-3e43-4847-9573-ac0e1b9dbef7\" (UID: \"cc9aa361-3e43-4847-9573-ac0e1b9dbef7\") "
Aug 04 18:34:15.757305 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:34:15.757186    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc9aa361-3e43-4847-9573-ac0e1b9dbef7-tls-certs\") pod \"cc9aa361-3e43-4847-9573-ac0e1b9dbef7\" (UID: \"cc9aa361-3e43-4847-9573-ac0e1b9dbef7\") "
Aug 04 18:34:15.757305 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:34:15.757233    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc9aa361-3e43-4847-9573-ac0e1b9dbef7-dshm\") pod \"cc9aa361-3e43-4847-9573-ac0e1b9dbef7\" (UID: \"cc9aa361-3e43-4847-9573-ac0e1b9dbef7\") "
Aug 04 18:34:15.757305 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:34:15.757272    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc9aa361-3e43-4847-9573-ac0e1b9dbef7-model-cache\") pod \"cc9aa361-3e43-4847-9573-ac0e1b9dbef7\" (UID: \"cc9aa361-3e43-4847-9573-ac0e1b9dbef7\") "
Aug 04 18:34:15.757606 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:34:15.757371    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc9aa361-3e43-4847-9573-ac0e1b9dbef7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "cc9aa361-3e43-4847-9573-ac0e1b9dbef7" (UID: "cc9aa361-3e43-4847-9573-ac0e1b9dbef7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:34:15.757606 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:34:15.757451    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc9aa361-3e43-4847-9573-ac0e1b9dbef7-home" (OuterVolumeSpecName: "home") pod "cc9aa361-3e43-4847-9573-ac0e1b9dbef7" (UID: "cc9aa361-3e43-4847-9573-ac0e1b9dbef7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:34:15.757606 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:34:15.757565    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc9aa361-3e43-4847-9573-ac0e1b9dbef7-tmp-dir\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:34:15.757606 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:34:15.757590    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc9aa361-3e43-4847-9573-ac0e1b9dbef7-home\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:34:15.757768 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:34:15.757625    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc9aa361-3e43-4847-9573-ac0e1b9dbef7-model-cache" (OuterVolumeSpecName: "model-cache") pod "cc9aa361-3e43-4847-9573-ac0e1b9dbef7" (UID: "cc9aa361-3e43-4847-9573-ac0e1b9dbef7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:34:15.759455 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:34:15.759424    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc9aa361-3e43-4847-9573-ac0e1b9dbef7-kube-api-access-xrqpb" (OuterVolumeSpecName: "kube-api-access-xrqpb") pod "cc9aa361-3e43-4847-9573-ac0e1b9dbef7" (UID: "cc9aa361-3e43-4847-9573-ac0e1b9dbef7"). InnerVolumeSpecName "kube-api-access-xrqpb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:34:15.759715 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:34:15.759696    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc9aa361-3e43-4847-9573-ac0e1b9dbef7-dshm" (OuterVolumeSpecName: "dshm") pod "cc9aa361-3e43-4847-9573-ac0e1b9dbef7" (UID: "cc9aa361-3e43-4847-9573-ac0e1b9dbef7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:34:15.759778 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:34:15.759712    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc9aa361-3e43-4847-9573-ac0e1b9dbef7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "cc9aa361-3e43-4847-9573-ac0e1b9dbef7" (UID: "cc9aa361-3e43-4847-9573-ac0e1b9dbef7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:34:15.858508 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:34:15.858449    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc9aa361-3e43-4847-9573-ac0e1b9dbef7-model-cache\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:34:15.858508 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:34:15.858504    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xrqpb\" (UniqueName: \"kubernetes.io/projected/cc9aa361-3e43-4847-9573-ac0e1b9dbef7-kube-api-access-xrqpb\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:34:15.858508 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:34:15.858514    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc9aa361-3e43-4847-9573-ac0e1b9dbef7-tls-certs\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:34:15.858748 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:34:15.858522    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc9aa361-3e43-4847-9573-ac0e1b9dbef7-dshm\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:34:16.342645 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:34:16.342611    2583 generic.go:356] "Generic (PLEG): container finished" podID="cc9aa361-3e43-4847-9573-ac0e1b9dbef7" containerID="39febdf09b18757638a7482bfe0f93657dc44d9887a2cdb64d69770b235209d0" exitCode=0
Aug 04 18:34:16.342836 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:34:16.342653    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-n92sg"
Aug 04 18:34:16.342836 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:34:16.342692    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-n92sg" event={"ID":"cc9aa361-3e43-4847-9573-ac0e1b9dbef7","Type":"ContainerDied","Data":"39febdf09b18757638a7482bfe0f93657dc44d9887a2cdb64d69770b235209d0"}
Aug 04 18:34:16.342836 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:34:16.342742    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-n92sg" event={"ID":"cc9aa361-3e43-4847-9573-ac0e1b9dbef7","Type":"ContainerDied","Data":"51605489e48f94bc2df90addfb65b889026edfe548e2210fa8a494212c51cfed"}
Aug 04 18:34:16.342836 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:34:16.342764    2583 scope.go:117] "RemoveContainer" containerID="39febdf09b18757638a7482bfe0f93657dc44d9887a2cdb64d69770b235209d0"
Aug 04 18:34:16.352665 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:34:16.352640    2583 scope.go:117] "RemoveContainer" containerID="39febdf09b18757638a7482bfe0f93657dc44d9887a2cdb64d69770b235209d0"
Aug 04 18:34:16.352970 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:34:16.352948    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39febdf09b18757638a7482bfe0f93657dc44d9887a2cdb64d69770b235209d0\": container with ID starting with 39febdf09b18757638a7482bfe0f93657dc44d9887a2cdb64d69770b235209d0 not found: ID does not exist" containerID="39febdf09b18757638a7482bfe0f93657dc44d9887a2cdb64d69770b235209d0"
Aug 04 18:34:16.353047 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:34:16.352983    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39febdf09b18757638a7482bfe0f93657dc44d9887a2cdb64d69770b235209d0"} err="failed to get container status \"39febdf09b18757638a7482bfe0f93657dc44d9887a2cdb64d69770b235209d0\": rpc error: code = NotFound desc = could not find container \"39febdf09b18757638a7482bfe0f93657dc44d9887a2cdb64d69770b235209d0\": container with ID starting with 39febdf09b18757638a7482bfe0f93657dc44d9887a2cdb64d69770b235209d0 not found: ID does not exist"
Aug 04 18:34:16.367664 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:34:16.367641    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-n92sg"]
Aug 04 18:34:16.373282 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:34:16.373249    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-n92sg"]
Aug 04 18:34:16.942679 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:34:16.942642    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cc9aa361-3e43-4847-9573-ac0e1b9dbef7" path="/var/lib/kubelet/pods/cc9aa361-3e43-4847-9573-ac0e1b9dbef7/volumes"
Aug 04 18:34:26.319109 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:34:26.319074    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-bz7tg"]
Aug 04 18:34:26.319506 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:34:26.319454    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc9aa361-3e43-4847-9573-ac0e1b9dbef7" containerName="main"
Aug 04 18:34:26.319506 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:34:26.319465    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc9aa361-3e43-4847-9573-ac0e1b9dbef7" containerName="main"
Aug 04 18:34:26.319506 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:34:26.319496    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b8fcf12-6c17-4f06-97c5-a54d458d79b4" containerName="main"
Aug 04 18:34:26.319506 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:34:26.319502    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b8fcf12-6c17-4f06-97c5-a54d458d79b4" containerName="main"
Aug 04 18:34:26.319652 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:34:26.319573    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc9aa361-3e43-4847-9573-ac0e1b9dbef7" containerName="main"
Aug 04 18:34:26.319652 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:34:26.319581    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="4b8fcf12-6c17-4f06-97c5-a54d458d79b4" containerName="main"
Aug 04 18:34:26.323206 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:34:26.323185    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-bz7tg"
Aug 04 18:34:26.326089 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:34:26.326064    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-three-member-group-c33d828b\"/\"tri-v1-kserve-self-signed-certs\""
Aug 04 18:34:26.328107 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:34:26.328085    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-three-member-group-c33d828b\"/\"openshift-service-ca.crt\""
Aug 04 18:34:26.328220 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:34:26.328117    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-three-member-group-c33d828b\"/\"kube-root-ca.crt\""
Aug 04 18:34:26.328220 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:34:26.328130    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-three-member-group-c33d828b\"/\"default-dockercfg-j6jrf\""
Aug 04 18:34:26.336306 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:34:26.336278    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-bz7tg"]
Aug 04 18:34:26.346891 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:34:26.346863    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc23c596-3a67-46b6-8d23-ab81151e0c5a-dshm\") pod \"tri-v1-kserve-79f8f98bc5-bz7tg\" (UID: \"dc23c596-3a67-46b6-8d23-ab81151e0c5a\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-bz7tg"
Aug 04 18:34:26.347005 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:34:26.346901    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc23c596-3a67-46b6-8d23-ab81151e0c5a-home\") pod \"tri-v1-kserve-79f8f98bc5-bz7tg\" (UID: \"dc23c596-3a67-46b6-8d23-ab81151e0c5a\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-bz7tg"
Aug 04 18:34:26.347005 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:34:26.346964    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc23c596-3a67-46b6-8d23-ab81151e0c5a-model-cache\") pod \"tri-v1-kserve-79f8f98bc5-bz7tg\" (UID: \"dc23c596-3a67-46b6-8d23-ab81151e0c5a\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-bz7tg"
Aug 04 18:34:26.347005 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:34:26.346993    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc23c596-3a67-46b6-8d23-ab81151e0c5a-tls-certs\") pod \"tri-v1-kserve-79f8f98bc5-bz7tg\" (UID: \"dc23c596-3a67-46b6-8d23-ab81151e0c5a\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-bz7tg"
Aug 04 18:34:26.347116 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:34:26.347041    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ncfh\" (UniqueName: \"kubernetes.io/projected/dc23c596-3a67-46b6-8d23-ab81151e0c5a-kube-api-access-7ncfh\") pod \"tri-v1-kserve-79f8f98bc5-bz7tg\" (UID: \"dc23c596-3a67-46b6-8d23-ab81151e0c5a\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-bz7tg"
Aug 04 18:34:26.347116 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:34:26.347093    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc23c596-3a67-46b6-8d23-ab81151e0c5a-tmp-dir\") pod \"tri-v1-kserve-79f8f98bc5-bz7tg\" (UID: \"dc23c596-3a67-46b6-8d23-ab81151e0c5a\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-bz7tg"
Aug 04 18:34:26.448278 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:34:26.448237    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc23c596-3a67-46b6-8d23-ab81151e0c5a-model-cache\") pod \"tri-v1-kserve-79f8f98bc5-bz7tg\" (UID: \"dc23c596-3a67-46b6-8d23-ab81151e0c5a\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-bz7tg"
Aug 04 18:34:26.448511 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:34:26.448299    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc23c596-3a67-46b6-8d23-ab81151e0c5a-tls-certs\") pod \"tri-v1-kserve-79f8f98bc5-bz7tg\" (UID: \"dc23c596-3a67-46b6-8d23-ab81151e0c5a\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-bz7tg"
Aug 04 18:34:26.448511 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:34:26.448320    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ncfh\" (UniqueName: \"kubernetes.io/projected/dc23c596-3a67-46b6-8d23-ab81151e0c5a-kube-api-access-7ncfh\") pod \"tri-v1-kserve-79f8f98bc5-bz7tg\" (UID: \"dc23c596-3a67-46b6-8d23-ab81151e0c5a\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-bz7tg"
Aug 04 18:34:26.448511 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:34:26.448377    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc23c596-3a67-46b6-8d23-ab81151e0c5a-tmp-dir\") pod \"tri-v1-kserve-79f8f98bc5-bz7tg\" (UID: \"dc23c596-3a67-46b6-8d23-ab81151e0c5a\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-bz7tg"
Aug 04 18:34:26.448511 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:34:26.448439    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc23c596-3a67-46b6-8d23-ab81151e0c5a-dshm\") pod \"tri-v1-kserve-79f8f98bc5-bz7tg\" (UID: \"dc23c596-3a67-46b6-8d23-ab81151e0c5a\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-bz7tg"
Aug 04 18:34:26.448511 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:34:26.448499    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc23c596-3a67-46b6-8d23-ab81151e0c5a-home\") pod \"tri-v1-kserve-79f8f98bc5-bz7tg\" (UID: \"dc23c596-3a67-46b6-8d23-ab81151e0c5a\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-bz7tg"
Aug 04 18:34:26.448948 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:34:26.448898    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc23c596-3a67-46b6-8d23-ab81151e0c5a-model-cache\") pod \"tri-v1-kserve-79f8f98bc5-bz7tg\" (UID: \"dc23c596-3a67-46b6-8d23-ab81151e0c5a\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-bz7tg"
Aug 04 18:34:26.449636 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:34:26.448942    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc23c596-3a67-46b6-8d23-ab81151e0c5a-tmp-dir\") pod \"tri-v1-kserve-79f8f98bc5-bz7tg\" (UID: \"dc23c596-3a67-46b6-8d23-ab81151e0c5a\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-bz7tg"
Aug 04 18:34:26.449636 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:34:26.448986    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc23c596-3a67-46b6-8d23-ab81151e0c5a-home\") pod \"tri-v1-kserve-79f8f98bc5-bz7tg\" (UID: \"dc23c596-3a67-46b6-8d23-ab81151e0c5a\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-bz7tg"
Aug 04 18:34:26.451156 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:34:26.451135    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc23c596-3a67-46b6-8d23-ab81151e0c5a-dshm\") pod \"tri-v1-kserve-79f8f98bc5-bz7tg\" (UID: \"dc23c596-3a67-46b6-8d23-ab81151e0c5a\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-bz7tg"
Aug 04 18:34:26.451252 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:34:26.451158    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc23c596-3a67-46b6-8d23-ab81151e0c5a-tls-certs\") pod \"tri-v1-kserve-79f8f98bc5-bz7tg\" (UID: \"dc23c596-3a67-46b6-8d23-ab81151e0c5a\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-bz7tg"
Aug 04 18:34:26.457742 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:34:26.457710    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ncfh\" (UniqueName: \"kubernetes.io/projected/dc23c596-3a67-46b6-8d23-ab81151e0c5a-kube-api-access-7ncfh\") pod \"tri-v1-kserve-79f8f98bc5-bz7tg\" (UID: \"dc23c596-3a67-46b6-8d23-ab81151e0c5a\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-bz7tg"
Aug 04 18:34:26.634696 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:34:26.634594    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-bz7tg"
Aug 04 18:34:26.782626 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:34:26.782588    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-bz7tg"]
Aug 04 18:34:26.783727 ip-10-0-135-124 kubenswrapper[2583]: W0804 18:34:26.783699    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc23c596_3a67_46b6_8d23_ab81151e0c5a.slice/crio-9f6e914ba5e0d78e48b7718375039119425958e3fcd31a6ac4481295604ae18a WatchSource:0}: Error finding container 9f6e914ba5e0d78e48b7718375039119425958e3fcd31a6ac4481295604ae18a: Status 404 returned error can't find the container with id 9f6e914ba5e0d78e48b7718375039119425958e3fcd31a6ac4481295604ae18a
Aug 04 18:34:27.410538 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:34:27.410494    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-bz7tg" event={"ID":"dc23c596-3a67-46b6-8d23-ab81151e0c5a","Type":"ContainerStarted","Data":"cc426ca43f0c6a80b7d4ec3c1da0c1de8338a738dd84195f66c8405a49495046"}
Aug 04 18:34:27.410919 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:34:27.410565    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-bz7tg" event={"ID":"dc23c596-3a67-46b6-8d23-ab81151e0c5a","Type":"ContainerStarted","Data":"9f6e914ba5e0d78e48b7718375039119425958e3fcd31a6ac4481295604ae18a"}
Aug 04 18:34:27.438330 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:34:27.438275    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-bz7tg" podStartSLOduration=1.438259625 podStartE2EDuration="1.438259625s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:34:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:34:27.437073562 +0000 UTC m=+2081.018292134" watchObservedRunningTime="2026-08-04 18:34:27.438259625 +0000 UTC m=+2081.019478258"
Aug 04 18:34:27.635611 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:34:27.635574    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-bz7tg"
Aug 04 18:34:33.317620 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:34:33.317588    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-jjbv6"]
Aug 04 18:34:33.321245 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:34:33.321226    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-jjbv6"
Aug 04 18:34:33.324580 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:34:33.324555    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-three-member-group-c33d828b\"/\"tri-v2-kserve-self-signed-certs\""
Aug 04 18:34:33.332618 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:34:33.332591    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-jjbv6"]
Aug 04 18:34:33.408648 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:34:33.408612    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7abb6fb0-53d5-452a-b2a2-6d3a38ee9fe1-home\") pod \"tri-v2-kserve-dcb554bf5-jjbv6\" (UID: \"7abb6fb0-53d5-452a-b2a2-6d3a38ee9fe1\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-jjbv6"
Aug 04 18:34:33.408648 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:34:33.408650    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7abb6fb0-53d5-452a-b2a2-6d3a38ee9fe1-dshm\") pod \"tri-v2-kserve-dcb554bf5-jjbv6\" (UID: \"7abb6fb0-53d5-452a-b2a2-6d3a38ee9fe1\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-jjbv6"
Aug 04 18:34:33.408880 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:34:33.408698    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7abb6fb0-53d5-452a-b2a2-6d3a38ee9fe1-model-cache\") pod \"tri-v2-kserve-dcb554bf5-jjbv6\" (UID: \"7abb6fb0-53d5-452a-b2a2-6d3a38ee9fe1\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-jjbv6"
Aug 04 18:34:33.408880 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:34:33.408721    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7abb6fb0-53d5-452a-b2a2-6d3a38ee9fe1-tmp-dir\") pod \"tri-v2-kserve-dcb554bf5-jjbv6\" (UID: \"7abb6fb0-53d5-452a-b2a2-6d3a38ee9fe1\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-jjbv6"
Aug 04 18:34:33.408880 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:34:33.408737    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlbgv\" (UniqueName: \"kubernetes.io/projected/7abb6fb0-53d5-452a-b2a2-6d3a38ee9fe1-kube-api-access-vlbgv\") pod \"tri-v2-kserve-dcb554bf5-jjbv6\" (UID: \"7abb6fb0-53d5-452a-b2a2-6d3a38ee9fe1\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-jjbv6"
Aug 04 18:34:33.408880 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:34:33.408760    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7abb6fb0-53d5-452a-b2a2-6d3a38ee9fe1-tls-certs\") pod \"tri-v2-kserve-dcb554bf5-jjbv6\" (UID: \"7abb6fb0-53d5-452a-b2a2-6d3a38ee9fe1\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-jjbv6"
Aug 04 18:34:33.510008 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:34:33.509961    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7abb6fb0-53d5-452a-b2a2-6d3a38ee9fe1-home\") pod \"tri-v2-kserve-dcb554bf5-jjbv6\" (UID: \"7abb6fb0-53d5-452a-b2a2-6d3a38ee9fe1\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-jjbv6"
Aug 04 18:34:33.510008 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:34:33.510012    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7abb6fb0-53d5-452a-b2a2-6d3a38ee9fe1-dshm\") pod \"tri-v2-kserve-dcb554bf5-jjbv6\" (UID: \"7abb6fb0-53d5-452a-b2a2-6d3a38ee9fe1\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-jjbv6"
Aug 04 18:34:33.510270 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:34:33.510069    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7abb6fb0-53d5-452a-b2a2-6d3a38ee9fe1-model-cache\") pod \"tri-v2-kserve-dcb554bf5-jjbv6\" (UID: \"7abb6fb0-53d5-452a-b2a2-6d3a38ee9fe1\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-jjbv6"
Aug 04 18:34:33.510270 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:34:33.510095    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7abb6fb0-53d5-452a-b2a2-6d3a38ee9fe1-tmp-dir\") pod \"tri-v2-kserve-dcb554bf5-jjbv6\" (UID: \"7abb6fb0-53d5-452a-b2a2-6d3a38ee9fe1\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-jjbv6"
Aug 04 18:34:33.510270 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:34:33.510116    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlbgv\" (UniqueName: \"kubernetes.io/projected/7abb6fb0-53d5-452a-b2a2-6d3a38ee9fe1-kube-api-access-vlbgv\") pod \"tri-v2-kserve-dcb554bf5-jjbv6\" (UID: \"7abb6fb0-53d5-452a-b2a2-6d3a38ee9fe1\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-jjbv6"
Aug 04 18:34:33.510270 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:34:33.510148    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7abb6fb0-53d5-452a-b2a2-6d3a38ee9fe1-tls-certs\") pod \"tri-v2-kserve-dcb554bf5-jjbv6\" (UID: \"7abb6fb0-53d5-452a-b2a2-6d3a38ee9fe1\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-jjbv6"
Aug 04 18:34:33.510519 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:34:33.510445    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7abb6fb0-53d5-452a-b2a2-6d3a38ee9fe1-home\") pod \"tri-v2-kserve-dcb554bf5-jjbv6\" (UID: \"7abb6fb0-53d5-452a-b2a2-6d3a38ee9fe1\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-jjbv6"
Aug 04 18:34:33.510519 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:34:33.510499    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7abb6fb0-53d5-452a-b2a2-6d3a38ee9fe1-tmp-dir\") pod \"tri-v2-kserve-dcb554bf5-jjbv6\" (UID: \"7abb6fb0-53d5-452a-b2a2-6d3a38ee9fe1\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-jjbv6"
Aug 04 18:34:33.510627 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:34:33.510567    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7abb6fb0-53d5-452a-b2a2-6d3a38ee9fe1-model-cache\") pod \"tri-v2-kserve-dcb554bf5-jjbv6\" (UID: \"7abb6fb0-53d5-452a-b2a2-6d3a38ee9fe1\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-jjbv6"
Aug 04 18:34:33.512413 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:34:33.512389    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7abb6fb0-53d5-452a-b2a2-6d3a38ee9fe1-dshm\") pod \"tri-v2-kserve-dcb554bf5-jjbv6\" (UID: \"7abb6fb0-53d5-452a-b2a2-6d3a38ee9fe1\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-jjbv6"
Aug 04 18:34:33.512555 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:34:33.512528    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7abb6fb0-53d5-452a-b2a2-6d3a38ee9fe1-tls-certs\") pod \"tri-v2-kserve-dcb554bf5-jjbv6\" (UID: \"7abb6fb0-53d5-452a-b2a2-6d3a38ee9fe1\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-jjbv6"
Aug 04 18:34:33.521149 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:34:33.521129    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlbgv\" (UniqueName: \"kubernetes.io/projected/7abb6fb0-53d5-452a-b2a2-6d3a38ee9fe1-kube-api-access-vlbgv\") pod \"tri-v2-kserve-dcb554bf5-jjbv6\" (UID: \"7abb6fb0-53d5-452a-b2a2-6d3a38ee9fe1\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-jjbv6"
Aug 04 18:34:33.633464 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:34:33.633354    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-jjbv6"
Aug 04 18:34:33.781055 ip-10-0-135-124 kubenswrapper[2583]: W0804 18:34:33.781020    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7abb6fb0_53d5_452a_b2a2_6d3a38ee9fe1.slice/crio-46fbd85892142b42686cda3b29e2fa7af91fd1e075ee7ec5df8a6d48b41af6fd WatchSource:0}: Error finding container 46fbd85892142b42686cda3b29e2fa7af91fd1e075ee7ec5df8a6d48b41af6fd: Status 404 returned error can't find the container with id 46fbd85892142b42686cda3b29e2fa7af91fd1e075ee7ec5df8a6d48b41af6fd
Aug 04 18:34:33.791012 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:34:33.790988    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-jjbv6"]
Aug 04 18:34:34.448239 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:34:34.448197    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-jjbv6" event={"ID":"7abb6fb0-53d5-452a-b2a2-6d3a38ee9fe1","Type":"ContainerStarted","Data":"6c632ed8d0141a83b62edd4698a0d7b90c6eaa0ac5a47f21140da59adbcd1c19"}
Aug 04 18:34:34.448741 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:34:34.448258    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-jjbv6" event={"ID":"7abb6fb0-53d5-452a-b2a2-6d3a38ee9fe1","Type":"ContainerStarted","Data":"46fbd85892142b42686cda3b29e2fa7af91fd1e075ee7ec5df8a6d48b41af6fd"}
Aug 04 18:34:34.469247 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:34:34.469192    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-jjbv6" podStartSLOduration=1.469174385 podStartE2EDuration="1.469174385s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:34:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:34:34.467707499 +0000 UTC m=+2088.048926076" watchObservedRunningTime="2026-08-04 18:34:34.469174385 +0000 UTC m=+2088.050392978"
Aug 04 18:34:34.634432 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:34:34.634381    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-jjbv6"
Aug 04 18:34:36.635074 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:34:36.635032    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-bz7tg"
Aug 04 18:34:36.647954 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:34:36.647916    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-bz7tg"
Aug 04 18:34:37.478119 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:34:37.478087    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-bz7tg"
Aug 04 18:34:43.634243 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:34:43.634192    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-jjbv6"
Aug 04 18:34:43.650173 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:34:43.650140    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-jjbv6"
Aug 04 18:34:44.515595 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:34:44.515563    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-jjbv6"
Aug 04 18:34:47.065333 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:34:47.065232    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-wskfb_6157c2d3-2fd0-4544-88ad-434ac75f8c04/console-operator/1.log"
Aug 04 18:34:47.081758 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:34:47.077232    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-wskfb_6157c2d3-2fd0-4544-88ad-434ac75f8c04/console-operator/1.log"
Aug 04 18:34:58.110277 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:34:58.110239    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-5b4mc"]
Aug 04 18:34:58.113896 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:34:58.113873    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-5b4mc"
Aug 04 18:34:58.116973 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:34:58.116955    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-three-member-group-c33d828b\"/\"tri-v3-kserve-self-signed-certs\""
Aug 04 18:34:58.130038 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:34:58.130010    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-5b4mc"]
Aug 04 18:34:58.241479 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:34:58.241443    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61d88969-cd9f-4def-9ca3-fa5ca9cdede7-tls-certs\") pod \"tri-v3-kserve-798c88c66c-5b4mc\" (UID: \"61d88969-cd9f-4def-9ca3-fa5ca9cdede7\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-5b4mc"
Aug 04 18:34:58.241664 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:34:58.241505    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5nk4q\" (UniqueName: \"kubernetes.io/projected/61d88969-cd9f-4def-9ca3-fa5ca9cdede7-kube-api-access-5nk4q\") pod \"tri-v3-kserve-798c88c66c-5b4mc\" (UID: \"61d88969-cd9f-4def-9ca3-fa5ca9cdede7\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-5b4mc"
Aug 04 18:34:58.241664 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:34:58.241638    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61d88969-cd9f-4def-9ca3-fa5ca9cdede7-model-cache\") pod \"tri-v3-kserve-798c88c66c-5b4mc\" (UID: \"61d88969-cd9f-4def-9ca3-fa5ca9cdede7\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-5b4mc"
Aug 04 18:34:58.241739 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:34:58.241722    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/61d88969-cd9f-4def-9ca3-fa5ca9cdede7-dshm\") pod \"tri-v3-kserve-798c88c66c-5b4mc\" (UID: \"61d88969-cd9f-4def-9ca3-fa5ca9cdede7\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-5b4mc"
Aug 04 18:34:58.241774 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:34:58.241749    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/61d88969-cd9f-4def-9ca3-fa5ca9cdede7-home\") pod \"tri-v3-kserve-798c88c66c-5b4mc\" (UID: \"61d88969-cd9f-4def-9ca3-fa5ca9cdede7\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-5b4mc"
Aug 04 18:34:58.241823 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:34:58.241778    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61d88969-cd9f-4def-9ca3-fa5ca9cdede7-tmp-dir\") pod \"tri-v3-kserve-798c88c66c-5b4mc\" (UID: \"61d88969-cd9f-4def-9ca3-fa5ca9cdede7\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-5b4mc"
Aug 04 18:34:58.343103 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:34:58.343062    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61d88969-cd9f-4def-9ca3-fa5ca9cdede7-model-cache\") pod \"tri-v3-kserve-798c88c66c-5b4mc\" (UID: \"61d88969-cd9f-4def-9ca3-fa5ca9cdede7\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-5b4mc"
Aug 04 18:34:58.343284 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:34:58.343128    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/61d88969-cd9f-4def-9ca3-fa5ca9cdede7-dshm\") pod \"tri-v3-kserve-798c88c66c-5b4mc\" (UID: \"61d88969-cd9f-4def-9ca3-fa5ca9cdede7\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-5b4mc"
Aug 04 18:34:58.343284 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:34:58.343202    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/61d88969-cd9f-4def-9ca3-fa5ca9cdede7-home\") pod \"tri-v3-kserve-798c88c66c-5b4mc\" (UID: \"61d88969-cd9f-4def-9ca3-fa5ca9cdede7\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-5b4mc"
Aug 04 18:34:58.343284 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:34:58.343253    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61d88969-cd9f-4def-9ca3-fa5ca9cdede7-tmp-dir\") pod \"tri-v3-kserve-798c88c66c-5b4mc\" (UID: \"61d88969-cd9f-4def-9ca3-fa5ca9cdede7\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-5b4mc"
Aug 04 18:34:58.343418 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:34:58.343297    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61d88969-cd9f-4def-9ca3-fa5ca9cdede7-tls-certs\") pod \"tri-v3-kserve-798c88c66c-5b4mc\" (UID: \"61d88969-cd9f-4def-9ca3-fa5ca9cdede7\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-5b4mc"
Aug 04 18:34:58.343418 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:34:58.343323    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5nk4q\" (UniqueName: \"kubernetes.io/projected/61d88969-cd9f-4def-9ca3-fa5ca9cdede7-kube-api-access-5nk4q\") pod \"tri-v3-kserve-798c88c66c-5b4mc\" (UID: \"61d88969-cd9f-4def-9ca3-fa5ca9cdede7\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-5b4mc"
Aug 04 18:34:58.343585 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:34:58.343565    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61d88969-cd9f-4def-9ca3-fa5ca9cdede7-model-cache\") pod \"tri-v3-kserve-798c88c66c-5b4mc\" (UID: \"61d88969-cd9f-4def-9ca3-fa5ca9cdede7\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-5b4mc"
Aug 04 18:34:58.343682 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:34:58.343661    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/61d88969-cd9f-4def-9ca3-fa5ca9cdede7-home\") pod \"tri-v3-kserve-798c88c66c-5b4mc\" (UID: \"61d88969-cd9f-4def-9ca3-fa5ca9cdede7\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-5b4mc"
Aug 04 18:34:58.343750 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:34:58.343670    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61d88969-cd9f-4def-9ca3-fa5ca9cdede7-tmp-dir\") pod \"tri-v3-kserve-798c88c66c-5b4mc\" (UID: \"61d88969-cd9f-4def-9ca3-fa5ca9cdede7\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-5b4mc"
Aug 04 18:34:58.345613 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:34:58.345582    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/61d88969-cd9f-4def-9ca3-fa5ca9cdede7-dshm\") pod \"tri-v3-kserve-798c88c66c-5b4mc\" (UID: \"61d88969-cd9f-4def-9ca3-fa5ca9cdede7\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-5b4mc"
Aug 04 18:34:58.345865 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:34:58.345849    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61d88969-cd9f-4def-9ca3-fa5ca9cdede7-tls-certs\") pod \"tri-v3-kserve-798c88c66c-5b4mc\" (UID: \"61d88969-cd9f-4def-9ca3-fa5ca9cdede7\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-5b4mc"
Aug 04 18:34:58.353427 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:34:58.353386    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5nk4q\" (UniqueName: \"kubernetes.io/projected/61d88969-cd9f-4def-9ca3-fa5ca9cdede7-kube-api-access-5nk4q\") pod \"tri-v3-kserve-798c88c66c-5b4mc\" (UID: \"61d88969-cd9f-4def-9ca3-fa5ca9cdede7\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-5b4mc"
Aug 04 18:34:58.424655 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:34:58.424553    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-5b4mc"
Aug 04 18:34:58.571087 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:34:58.571060    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-5b4mc"]
Aug 04 18:34:58.574071 ip-10-0-135-124 kubenswrapper[2583]: W0804 18:34:58.574040    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61d88969_cd9f_4def_9ca3_fa5ca9cdede7.slice/crio-f1e68215936bd7f364db1ed4dbbd2d74b15c21dd9ea3a8f1c8dccd0baaf30ead WatchSource:0}: Error finding container f1e68215936bd7f364db1ed4dbbd2d74b15c21dd9ea3a8f1c8dccd0baaf30ead: Status 404 returned error can't find the container with id f1e68215936bd7f364db1ed4dbbd2d74b15c21dd9ea3a8f1c8dccd0baaf30ead
Aug 04 18:34:58.588498 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:34:58.588451    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-5b4mc" event={"ID":"61d88969-cd9f-4def-9ca3-fa5ca9cdede7","Type":"ContainerStarted","Data":"f1e68215936bd7f364db1ed4dbbd2d74b15c21dd9ea3a8f1c8dccd0baaf30ead"}
Aug 04 18:34:59.595232 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:34:59.595191    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-5b4mc" event={"ID":"61d88969-cd9f-4def-9ca3-fa5ca9cdede7","Type":"ContainerStarted","Data":"0b2f224ab659c09fc3e811e7c1b31d3117124e63f504906a24553bb2afd6d979"}
Aug 04 18:34:59.630105 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:34:59.630055    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-5b4mc" podStartSLOduration=1.6300413379999998 podStartE2EDuration="1.630041338s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:34:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:34:59.628458664 +0000 UTC m=+2113.209677242" watchObservedRunningTime="2026-08-04 18:34:59.630041338 +0000 UTC m=+2113.211259917"
Aug 04 18:35:00.424967 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:35:00.424928    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-5b4mc"
Aug 04 18:35:08.425308 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:35:08.425270    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-5b4mc"
Aug 04 18:35:08.438116 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:35:08.438087    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-5b4mc"
Aug 04 18:35:08.657856 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:35:08.657826    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-5b4mc"
Aug 04 18:36:12.634105 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:36:12.634058    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-jjbv6"]
Aug 04 18:36:13.531250 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:36:13.531218    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-5b4mc"]
Aug 04 18:36:16.787615 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:36:16.787576    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-bz7tg"]
Aug 04 18:36:27.653866 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:36:27.653823    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-jjbv6" podUID="7abb6fb0-53d5-452a-b2a2-6d3a38ee9fe1" containerName="main" containerID="cri-o://6c632ed8d0141a83b62edd4698a0d7b90c6eaa0ac5a47f21140da59adbcd1c19" gracePeriod=45
Aug 04 18:36:27.899261 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:36:27.899234    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-jjbv6"
Aug 04 18:36:28.034963 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:36:28.034910    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7abb6fb0-53d5-452a-b2a2-6d3a38ee9fe1-home\") pod \"7abb6fb0-53d5-452a-b2a2-6d3a38ee9fe1\" (UID: \"7abb6fb0-53d5-452a-b2a2-6d3a38ee9fe1\") "
Aug 04 18:36:28.035158 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:36:28.034978    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vlbgv\" (UniqueName: \"kubernetes.io/projected/7abb6fb0-53d5-452a-b2a2-6d3a38ee9fe1-kube-api-access-vlbgv\") pod \"7abb6fb0-53d5-452a-b2a2-6d3a38ee9fe1\" (UID: \"7abb6fb0-53d5-452a-b2a2-6d3a38ee9fe1\") "
Aug 04 18:36:28.035158 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:36:28.035039    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7abb6fb0-53d5-452a-b2a2-6d3a38ee9fe1-dshm\") pod \"7abb6fb0-53d5-452a-b2a2-6d3a38ee9fe1\" (UID: \"7abb6fb0-53d5-452a-b2a2-6d3a38ee9fe1\") "
Aug 04 18:36:28.035158 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:36:28.035111    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7abb6fb0-53d5-452a-b2a2-6d3a38ee9fe1-model-cache\") pod \"7abb6fb0-53d5-452a-b2a2-6d3a38ee9fe1\" (UID: \"7abb6fb0-53d5-452a-b2a2-6d3a38ee9fe1\") "
Aug 04 18:36:28.035342 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:36:28.035169    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7abb6fb0-53d5-452a-b2a2-6d3a38ee9fe1-tmp-dir\") pod \"7abb6fb0-53d5-452a-b2a2-6d3a38ee9fe1\" (UID: \"7abb6fb0-53d5-452a-b2a2-6d3a38ee9fe1\") "
Aug 04 18:36:28.035342 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:36:28.035207    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7abb6fb0-53d5-452a-b2a2-6d3a38ee9fe1-tls-certs\") pod \"7abb6fb0-53d5-452a-b2a2-6d3a38ee9fe1\" (UID: \"7abb6fb0-53d5-452a-b2a2-6d3a38ee9fe1\") "
Aug 04 18:36:28.035342 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:36:28.035250    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7abb6fb0-53d5-452a-b2a2-6d3a38ee9fe1-home" (OuterVolumeSpecName: "home") pod "7abb6fb0-53d5-452a-b2a2-6d3a38ee9fe1" (UID: "7abb6fb0-53d5-452a-b2a2-6d3a38ee9fe1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:36:28.035510 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:36:28.035346    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7abb6fb0-53d5-452a-b2a2-6d3a38ee9fe1-model-cache" (OuterVolumeSpecName: "model-cache") pod "7abb6fb0-53d5-452a-b2a2-6d3a38ee9fe1" (UID: "7abb6fb0-53d5-452a-b2a2-6d3a38ee9fe1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:36:28.035510 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:36:28.035425    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7abb6fb0-53d5-452a-b2a2-6d3a38ee9fe1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7abb6fb0-53d5-452a-b2a2-6d3a38ee9fe1" (UID: "7abb6fb0-53d5-452a-b2a2-6d3a38ee9fe1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:36:28.035746 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:36:28.035724    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7abb6fb0-53d5-452a-b2a2-6d3a38ee9fe1-model-cache\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:36:28.035827 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:36:28.035750    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7abb6fb0-53d5-452a-b2a2-6d3a38ee9fe1-tmp-dir\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:36:28.035827 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:36:28.035764    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7abb6fb0-53d5-452a-b2a2-6d3a38ee9fe1-home\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:36:28.037348 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:36:28.037323    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7abb6fb0-53d5-452a-b2a2-6d3a38ee9fe1-dshm" (OuterVolumeSpecName: "dshm") pod "7abb6fb0-53d5-452a-b2a2-6d3a38ee9fe1" (UID: "7abb6fb0-53d5-452a-b2a2-6d3a38ee9fe1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:36:28.037348 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:36:28.037339    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7abb6fb0-53d5-452a-b2a2-6d3a38ee9fe1-kube-api-access-vlbgv" (OuterVolumeSpecName: "kube-api-access-vlbgv") pod "7abb6fb0-53d5-452a-b2a2-6d3a38ee9fe1" (UID: "7abb6fb0-53d5-452a-b2a2-6d3a38ee9fe1"). InnerVolumeSpecName "kube-api-access-vlbgv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:36:28.037533 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:36:28.037483    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7abb6fb0-53d5-452a-b2a2-6d3a38ee9fe1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7abb6fb0-53d5-452a-b2a2-6d3a38ee9fe1" (UID: "7abb6fb0-53d5-452a-b2a2-6d3a38ee9fe1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:36:28.084459 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:36:28.084424    2583 generic.go:356] "Generic (PLEG): container finished" podID="7abb6fb0-53d5-452a-b2a2-6d3a38ee9fe1" containerID="6c632ed8d0141a83b62edd4698a0d7b90c6eaa0ac5a47f21140da59adbcd1c19" exitCode=0
Aug 04 18:36:28.084590 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:36:28.084486    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-jjbv6"
Aug 04 18:36:28.084590 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:36:28.084500    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-jjbv6" event={"ID":"7abb6fb0-53d5-452a-b2a2-6d3a38ee9fe1","Type":"ContainerDied","Data":"6c632ed8d0141a83b62edd4698a0d7b90c6eaa0ac5a47f21140da59adbcd1c19"}
Aug 04 18:36:28.084590 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:36:28.084556    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-jjbv6" event={"ID":"7abb6fb0-53d5-452a-b2a2-6d3a38ee9fe1","Type":"ContainerDied","Data":"46fbd85892142b42686cda3b29e2fa7af91fd1e075ee7ec5df8a6d48b41af6fd"}
Aug 04 18:36:28.084590 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:36:28.084578    2583 scope.go:117] "RemoveContainer" containerID="6c632ed8d0141a83b62edd4698a0d7b90c6eaa0ac5a47f21140da59adbcd1c19"
Aug 04 18:36:28.094337 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:36:28.094319    2583 scope.go:117] "RemoveContainer" containerID="6c632ed8d0141a83b62edd4698a0d7b90c6eaa0ac5a47f21140da59adbcd1c19"
Aug 04 18:36:28.094608 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:36:28.094589    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c632ed8d0141a83b62edd4698a0d7b90c6eaa0ac5a47f21140da59adbcd1c19\": container with ID starting with 6c632ed8d0141a83b62edd4698a0d7b90c6eaa0ac5a47f21140da59adbcd1c19 not found: ID does not exist" containerID="6c632ed8d0141a83b62edd4698a0d7b90c6eaa0ac5a47f21140da59adbcd1c19"
Aug 04 18:36:28.094677 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:36:28.094621    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c632ed8d0141a83b62edd4698a0d7b90c6eaa0ac5a47f21140da59adbcd1c19"} err="failed to get container status \"6c632ed8d0141a83b62edd4698a0d7b90c6eaa0ac5a47f21140da59adbcd1c19\": rpc error: code = NotFound desc = could not find container \"6c632ed8d0141a83b62edd4698a0d7b90c6eaa0ac5a47f21140da59adbcd1c19\": container with ID starting with 6c632ed8d0141a83b62edd4698a0d7b90c6eaa0ac5a47f21140da59adbcd1c19 not found: ID does not exist"
Aug 04 18:36:28.108690 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:36:28.108668    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-jjbv6"]
Aug 04 18:36:28.113233 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:36:28.113204    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-jjbv6"]
Aug 04 18:36:28.137110 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:36:28.137091    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7abb6fb0-53d5-452a-b2a2-6d3a38ee9fe1-tls-certs\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:36:28.137110 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:36:28.137110    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vlbgv\" (UniqueName: \"kubernetes.io/projected/7abb6fb0-53d5-452a-b2a2-6d3a38ee9fe1-kube-api-access-vlbgv\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:36:28.137238 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:36:28.137120    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7abb6fb0-53d5-452a-b2a2-6d3a38ee9fe1-dshm\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:36:28.550169 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:36:28.550131    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-5b4mc" podUID="61d88969-cd9f-4def-9ca3-fa5ca9cdede7" containerName="main" containerID="cri-o://0b2f224ab659c09fc3e811e7c1b31d3117124e63f504906a24553bb2afd6d979" gracePeriod=45
Aug 04 18:36:28.647713 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:36:28.647674    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-5b4mc" podUID="61d88969-cd9f-4def-9ca3-fa5ca9cdede7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": dial tcp 10.132.0.73:8000: connect: connection refused"
Aug 04 18:36:28.798998 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:36:28.798972    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-5b4mc"
Aug 04 18:36:28.942274 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:36:28.942243    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7abb6fb0-53d5-452a-b2a2-6d3a38ee9fe1" path="/var/lib/kubelet/pods/7abb6fb0-53d5-452a-b2a2-6d3a38ee9fe1/volumes"
Aug 04 18:36:28.943992 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:36:28.943972    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/61d88969-cd9f-4def-9ca3-fa5ca9cdede7-dshm\") pod \"61d88969-cd9f-4def-9ca3-fa5ca9cdede7\" (UID: \"61d88969-cd9f-4def-9ca3-fa5ca9cdede7\") "
Aug 04 18:36:28.944057 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:36:28.944016    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5nk4q\" (UniqueName: \"kubernetes.io/projected/61d88969-cd9f-4def-9ca3-fa5ca9cdede7-kube-api-access-5nk4q\") pod \"61d88969-cd9f-4def-9ca3-fa5ca9cdede7\" (UID: \"61d88969-cd9f-4def-9ca3-fa5ca9cdede7\") "
Aug 04 18:36:28.944104 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:36:28.944062    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61d88969-cd9f-4def-9ca3-fa5ca9cdede7-tls-certs\") pod \"61d88969-cd9f-4def-9ca3-fa5ca9cdede7\" (UID: \"61d88969-cd9f-4def-9ca3-fa5ca9cdede7\") "
Aug 04 18:36:28.944104 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:36:28.944077    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61d88969-cd9f-4def-9ca3-fa5ca9cdede7-tmp-dir\") pod \"61d88969-cd9f-4def-9ca3-fa5ca9cdede7\" (UID: \"61d88969-cd9f-4def-9ca3-fa5ca9cdede7\") "
Aug 04 18:36:28.944185 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:36:28.944112    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61d88969-cd9f-4def-9ca3-fa5ca9cdede7-model-cache\") pod \"61d88969-cd9f-4def-9ca3-fa5ca9cdede7\" (UID: \"61d88969-cd9f-4def-9ca3-fa5ca9cdede7\") "
Aug 04 18:36:28.944185 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:36:28.944128    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/61d88969-cd9f-4def-9ca3-fa5ca9cdede7-home\") pod \"61d88969-cd9f-4def-9ca3-fa5ca9cdede7\" (UID: \"61d88969-cd9f-4def-9ca3-fa5ca9cdede7\") "
Aug 04 18:36:28.944399 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:36:28.944371    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61d88969-cd9f-4def-9ca3-fa5ca9cdede7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "61d88969-cd9f-4def-9ca3-fa5ca9cdede7" (UID: "61d88969-cd9f-4def-9ca3-fa5ca9cdede7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:36:28.944525 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:36:28.944379    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61d88969-cd9f-4def-9ca3-fa5ca9cdede7-model-cache" (OuterVolumeSpecName: "model-cache") pod "61d88969-cd9f-4def-9ca3-fa5ca9cdede7" (UID: "61d88969-cd9f-4def-9ca3-fa5ca9cdede7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:36:28.944525 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:36:28.944455    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61d88969-cd9f-4def-9ca3-fa5ca9cdede7-home" (OuterVolumeSpecName: "home") pod "61d88969-cd9f-4def-9ca3-fa5ca9cdede7" (UID: "61d88969-cd9f-4def-9ca3-fa5ca9cdede7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:36:28.946164 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:36:28.946141    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61d88969-cd9f-4def-9ca3-fa5ca9cdede7-dshm" (OuterVolumeSpecName: "dshm") pod "61d88969-cd9f-4def-9ca3-fa5ca9cdede7" (UID: "61d88969-cd9f-4def-9ca3-fa5ca9cdede7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:36:28.946271 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:36:28.946184    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61d88969-cd9f-4def-9ca3-fa5ca9cdede7-kube-api-access-5nk4q" (OuterVolumeSpecName: "kube-api-access-5nk4q") pod "61d88969-cd9f-4def-9ca3-fa5ca9cdede7" (UID: "61d88969-cd9f-4def-9ca3-fa5ca9cdede7"). InnerVolumeSpecName "kube-api-access-5nk4q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:36:28.946271 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:36:28.946212    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61d88969-cd9f-4def-9ca3-fa5ca9cdede7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "61d88969-cd9f-4def-9ca3-fa5ca9cdede7" (UID: "61d88969-cd9f-4def-9ca3-fa5ca9cdede7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:36:29.045687 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:36:29.045653    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/61d88969-cd9f-4def-9ca3-fa5ca9cdede7-dshm\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:36:29.045687 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:36:29.045685    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5nk4q\" (UniqueName: \"kubernetes.io/projected/61d88969-cd9f-4def-9ca3-fa5ca9cdede7-kube-api-access-5nk4q\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:36:29.045687 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:36:29.045696    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61d88969-cd9f-4def-9ca3-fa5ca9cdede7-tls-certs\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:36:29.045925 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:36:29.045705    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61d88969-cd9f-4def-9ca3-fa5ca9cdede7-tmp-dir\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:36:29.045925 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:36:29.045713    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61d88969-cd9f-4def-9ca3-fa5ca9cdede7-model-cache\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:36:29.045925 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:36:29.045720    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/61d88969-cd9f-4def-9ca3-fa5ca9cdede7-home\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:36:29.092353 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:36:29.092321    2583 generic.go:356] "Generic (PLEG): container finished" podID="61d88969-cd9f-4def-9ca3-fa5ca9cdede7" containerID="0b2f224ab659c09fc3e811e7c1b31d3117124e63f504906a24553bb2afd6d979" exitCode=0
Aug 04 18:36:29.092519 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:36:29.092361    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-5b4mc"
Aug 04 18:36:29.092519 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:36:29.092397    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-5b4mc" event={"ID":"61d88969-cd9f-4def-9ca3-fa5ca9cdede7","Type":"ContainerDied","Data":"0b2f224ab659c09fc3e811e7c1b31d3117124e63f504906a24553bb2afd6d979"}
Aug 04 18:36:29.092519 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:36:29.092441    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-5b4mc" event={"ID":"61d88969-cd9f-4def-9ca3-fa5ca9cdede7","Type":"ContainerDied","Data":"f1e68215936bd7f364db1ed4dbbd2d74b15c21dd9ea3a8f1c8dccd0baaf30ead"}
Aug 04 18:36:29.092519 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:36:29.092489    2583 scope.go:117] "RemoveContainer" containerID="0b2f224ab659c09fc3e811e7c1b31d3117124e63f504906a24553bb2afd6d979"
Aug 04 18:36:29.109611 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:36:29.109590    2583 scope.go:117] "RemoveContainer" containerID="0b2f224ab659c09fc3e811e7c1b31d3117124e63f504906a24553bb2afd6d979"
Aug 04 18:36:29.109829 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:36:29.109810    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b2f224ab659c09fc3e811e7c1b31d3117124e63f504906a24553bb2afd6d979\": container with ID starting with 0b2f224ab659c09fc3e811e7c1b31d3117124e63f504906a24553bb2afd6d979 not found: ID does not exist" containerID="0b2f224ab659c09fc3e811e7c1b31d3117124e63f504906a24553bb2afd6d979"
Aug 04 18:36:29.109889 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:36:29.109835    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b2f224ab659c09fc3e811e7c1b31d3117124e63f504906a24553bb2afd6d979"} err="failed to get container status \"0b2f224ab659c09fc3e811e7c1b31d3117124e63f504906a24553bb2afd6d979\": rpc error: code = NotFound desc = could not find container \"0b2f224ab659c09fc3e811e7c1b31d3117124e63f504906a24553bb2afd6d979\": container with ID starting with 0b2f224ab659c09fc3e811e7c1b31d3117124e63f504906a24553bb2afd6d979 not found: ID does not exist"
Aug 04 18:36:29.123016 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:36:29.122990    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-5b4mc"]
Aug 04 18:36:29.126916 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:36:29.126892    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-5b4mc"]
Aug 04 18:36:30.942169 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:36:30.942135    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="61d88969-cd9f-4def-9ca3-fa5ca9cdede7" path="/var/lib/kubelet/pods/61d88969-cd9f-4def-9ca3-fa5ca9cdede7/volumes"
Aug 04 18:36:31.514458 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:36:31.514417    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-brl65"]
Aug 04 18:36:31.514899 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:36:31.514881    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="61d88969-cd9f-4def-9ca3-fa5ca9cdede7" containerName="main"
Aug 04 18:36:31.514899 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:36:31.514900    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="61d88969-cd9f-4def-9ca3-fa5ca9cdede7" containerName="main"
Aug 04 18:36:31.515055 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:36:31.514923    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7abb6fb0-53d5-452a-b2a2-6d3a38ee9fe1" containerName="main"
Aug 04 18:36:31.515055 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:36:31.514931    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="7abb6fb0-53d5-452a-b2a2-6d3a38ee9fe1" containerName="main"
Aug 04 18:36:31.515142 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:36:31.515070    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="61d88969-cd9f-4def-9ca3-fa5ca9cdede7" containerName="main"
Aug 04 18:36:31.515142 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:36:31.515087    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="7abb6fb0-53d5-452a-b2a2-6d3a38ee9fe1" containerName="main"
Aug 04 18:36:31.519560 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:36:31.519538    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-brl65"
Aug 04 18:36:31.522560 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:36:31.522538    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-late-join-483adbac\"/\"kube-root-ca.crt\""
Aug 04 18:36:31.523811 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:36:31.523791    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-late-join-483adbac\"/\"late-v1-kserve-self-signed-certs\""
Aug 04 18:36:31.523908 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:36:31.523794    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-late-join-483adbac\"/\"default-dockercfg-bpt7h\""
Aug 04 18:36:31.523908 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:36:31.523877    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-late-join-483adbac\"/\"openshift-service-ca.crt\""
Aug 04 18:36:31.530897 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:36:31.530872    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-brl65"]
Aug 04 18:36:31.666526 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:36:31.666487    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6mb8c\" (UniqueName: \"kubernetes.io/projected/783c6495-c0e8-4cd3-8dae-b7bbca43c8e8-kube-api-access-6mb8c\") pod \"late-v1-kserve-56cbc8456c-brl65\" (UID: \"783c6495-c0e8-4cd3-8dae-b7bbca43c8e8\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-brl65"
Aug 04 18:36:31.666692 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:36:31.666542    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/783c6495-c0e8-4cd3-8dae-b7bbca43c8e8-tls-certs\") pod \"late-v1-kserve-56cbc8456c-brl65\" (UID: \"783c6495-c0e8-4cd3-8dae-b7bbca43c8e8\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-brl65"
Aug 04 18:36:31.666692 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:36:31.666571    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/783c6495-c0e8-4cd3-8dae-b7bbca43c8e8-dshm\") pod \"late-v1-kserve-56cbc8456c-brl65\" (UID: \"783c6495-c0e8-4cd3-8dae-b7bbca43c8e8\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-brl65"
Aug 04 18:36:31.666768 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:36:31.666695    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/783c6495-c0e8-4cd3-8dae-b7bbca43c8e8-model-cache\") pod \"late-v1-kserve-56cbc8456c-brl65\" (UID: \"783c6495-c0e8-4cd3-8dae-b7bbca43c8e8\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-brl65"
Aug 04 18:36:31.666768 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:36:31.666722    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/783c6495-c0e8-4cd3-8dae-b7bbca43c8e8-home\") pod \"late-v1-kserve-56cbc8456c-brl65\" (UID: \"783c6495-c0e8-4cd3-8dae-b7bbca43c8e8\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-brl65"
Aug 04 18:36:31.666768 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:36:31.666740    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/783c6495-c0e8-4cd3-8dae-b7bbca43c8e8-tmp-dir\") pod \"late-v1-kserve-56cbc8456c-brl65\" (UID: \"783c6495-c0e8-4cd3-8dae-b7bbca43c8e8\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-brl65"
Aug 04 18:36:31.767882 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:36:31.767773    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/783c6495-c0e8-4cd3-8dae-b7bbca43c8e8-tmp-dir\") pod \"late-v1-kserve-56cbc8456c-brl65\" (UID: \"783c6495-c0e8-4cd3-8dae-b7bbca43c8e8\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-brl65"
Aug 04 18:36:31.767882 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:36:31.767823    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mb8c\" (UniqueName: \"kubernetes.io/projected/783c6495-c0e8-4cd3-8dae-b7bbca43c8e8-kube-api-access-6mb8c\") pod \"late-v1-kserve-56cbc8456c-brl65\" (UID: \"783c6495-c0e8-4cd3-8dae-b7bbca43c8e8\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-brl65"
Aug 04 18:36:31.767882 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:36:31.767872    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/783c6495-c0e8-4cd3-8dae-b7bbca43c8e8-tls-certs\") pod \"late-v1-kserve-56cbc8456c-brl65\" (UID: \"783c6495-c0e8-4cd3-8dae-b7bbca43c8e8\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-brl65"
Aug 04 18:36:31.768176 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:36:31.767901    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/783c6495-c0e8-4cd3-8dae-b7bbca43c8e8-dshm\") pod \"late-v1-kserve-56cbc8456c-brl65\" (UID: \"783c6495-c0e8-4cd3-8dae-b7bbca43c8e8\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-brl65"
Aug 04 18:36:31.768176 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:36:31.768003    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/783c6495-c0e8-4cd3-8dae-b7bbca43c8e8-model-cache\") pod \"late-v1-kserve-56cbc8456c-brl65\" (UID: \"783c6495-c0e8-4cd3-8dae-b7bbca43c8e8\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-brl65"
Aug 04 18:36:31.768176 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:36:31.768026    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/783c6495-c0e8-4cd3-8dae-b7bbca43c8e8-home\") pod \"late-v1-kserve-56cbc8456c-brl65\" (UID: \"783c6495-c0e8-4cd3-8dae-b7bbca43c8e8\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-brl65"
Aug 04 18:36:31.768336 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:36:31.768270    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/783c6495-c0e8-4cd3-8dae-b7bbca43c8e8-tmp-dir\") pod \"late-v1-kserve-56cbc8456c-brl65\" (UID: \"783c6495-c0e8-4cd3-8dae-b7bbca43c8e8\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-brl65"
Aug 04 18:36:31.768396 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:36:31.768372    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/783c6495-c0e8-4cd3-8dae-b7bbca43c8e8-model-cache\") pod \"late-v1-kserve-56cbc8456c-brl65\" (UID: \"783c6495-c0e8-4cd3-8dae-b7bbca43c8e8\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-brl65"
Aug 04 18:36:31.768485 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:36:31.768429    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/783c6495-c0e8-4cd3-8dae-b7bbca43c8e8-home\") pod \"late-v1-kserve-56cbc8456c-brl65\" (UID: \"783c6495-c0e8-4cd3-8dae-b7bbca43c8e8\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-brl65"
Aug 04 18:36:31.770287 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:36:31.770255    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/783c6495-c0e8-4cd3-8dae-b7bbca43c8e8-dshm\") pod \"late-v1-kserve-56cbc8456c-brl65\" (UID: \"783c6495-c0e8-4cd3-8dae-b7bbca43c8e8\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-brl65"
Aug 04 18:36:31.770562 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:36:31.770543    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/783c6495-c0e8-4cd3-8dae-b7bbca43c8e8-tls-certs\") pod \"late-v1-kserve-56cbc8456c-brl65\" (UID: \"783c6495-c0e8-4cd3-8dae-b7bbca43c8e8\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-brl65"
Aug 04 18:36:31.779075 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:36:31.779049    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mb8c\" (UniqueName: \"kubernetes.io/projected/783c6495-c0e8-4cd3-8dae-b7bbca43c8e8-kube-api-access-6mb8c\") pod \"late-v1-kserve-56cbc8456c-brl65\" (UID: \"783c6495-c0e8-4cd3-8dae-b7bbca43c8e8\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-brl65"
Aug 04 18:36:31.806856 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:36:31.806812    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-bz7tg" podUID="dc23c596-3a67-46b6-8d23-ab81151e0c5a" containerName="main" containerID="cri-o://cc426ca43f0c6a80b7d4ec3c1da0c1de8338a738dd84195f66c8405a49495046" gracePeriod=45
Aug 04 18:36:31.830277 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:36:31.830235    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-brl65"
Aug 04 18:36:31.981073 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:36:31.981047    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-brl65"]
Aug 04 18:36:31.982145 ip-10-0-135-124 kubenswrapper[2583]: W0804 18:36:31.982113    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod783c6495_c0e8_4cd3_8dae_b7bbca43c8e8.slice/crio-68b436b75ec0f708adb46a6baa03a3219e06740f4acd00f9b46f013417fcf79f WatchSource:0}: Error finding container 68b436b75ec0f708adb46a6baa03a3219e06740f4acd00f9b46f013417fcf79f: Status 404 returned error can't find the container with id 68b436b75ec0f708adb46a6baa03a3219e06740f4acd00f9b46f013417fcf79f
Aug 04 18:36:32.057353 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:36:32.057326    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-bz7tg"
Aug 04 18:36:32.112623 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:36:32.112586    2583 generic.go:356] "Generic (PLEG): container finished" podID="dc23c596-3a67-46b6-8d23-ab81151e0c5a" containerID="cc426ca43f0c6a80b7d4ec3c1da0c1de8338a738dd84195f66c8405a49495046" exitCode=0
Aug 04 18:36:32.112799 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:36:32.112642    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-bz7tg"
Aug 04 18:36:32.112799 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:36:32.112673    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-bz7tg" event={"ID":"dc23c596-3a67-46b6-8d23-ab81151e0c5a","Type":"ContainerDied","Data":"cc426ca43f0c6a80b7d4ec3c1da0c1de8338a738dd84195f66c8405a49495046"}
Aug 04 18:36:32.112799 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:36:32.112744    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-bz7tg" event={"ID":"dc23c596-3a67-46b6-8d23-ab81151e0c5a","Type":"ContainerDied","Data":"9f6e914ba5e0d78e48b7718375039119425958e3fcd31a6ac4481295604ae18a"}
Aug 04 18:36:32.112799 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:36:32.112777    2583 scope.go:117] "RemoveContainer" containerID="cc426ca43f0c6a80b7d4ec3c1da0c1de8338a738dd84195f66c8405a49495046"
Aug 04 18:36:32.114940 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:36:32.114914    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-brl65" event={"ID":"783c6495-c0e8-4cd3-8dae-b7bbca43c8e8","Type":"ContainerStarted","Data":"7294e4a1398960ac3202d771e535fa4fa451b1bec4211300707bcfaeb06cbbc4"}
Aug 04 18:36:32.115073 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:36:32.114962    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-brl65" event={"ID":"783c6495-c0e8-4cd3-8dae-b7bbca43c8e8","Type":"ContainerStarted","Data":"68b436b75ec0f708adb46a6baa03a3219e06740f4acd00f9b46f013417fcf79f"}
Aug 04 18:36:32.122069 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:36:32.122048    2583 scope.go:117] "RemoveContainer" containerID="cc426ca43f0c6a80b7d4ec3c1da0c1de8338a738dd84195f66c8405a49495046"
Aug 04 18:36:32.122353 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:36:32.122329    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc426ca43f0c6a80b7d4ec3c1da0c1de8338a738dd84195f66c8405a49495046\": container with ID starting with cc426ca43f0c6a80b7d4ec3c1da0c1de8338a738dd84195f66c8405a49495046 not found: ID does not exist" containerID="cc426ca43f0c6a80b7d4ec3c1da0c1de8338a738dd84195f66c8405a49495046"
Aug 04 18:36:32.122424 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:36:32.122360    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc426ca43f0c6a80b7d4ec3c1da0c1de8338a738dd84195f66c8405a49495046"} err="failed to get container status \"cc426ca43f0c6a80b7d4ec3c1da0c1de8338a738dd84195f66c8405a49495046\": rpc error: code = NotFound desc = could not find container \"cc426ca43f0c6a80b7d4ec3c1da0c1de8338a738dd84195f66c8405a49495046\": container with ID starting with cc426ca43f0c6a80b7d4ec3c1da0c1de8338a738dd84195f66c8405a49495046 not found: ID does not exist"
Aug 04 18:36:32.137252 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:36:32.137190    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-brl65" podStartSLOduration=1.137171192 podStartE2EDuration="1.137171192s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:36:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:36:32.134955435 +0000 UTC m=+2205.716174013" watchObservedRunningTime="2026-08-04 18:36:32.137171192 +0000 UTC m=+2205.718389770"
Aug 04 18:36:32.172523 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:36:32.172491    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc23c596-3a67-46b6-8d23-ab81151e0c5a-home\") pod \"dc23c596-3a67-46b6-8d23-ab81151e0c5a\" (UID: \"dc23c596-3a67-46b6-8d23-ab81151e0c5a\") "
Aug 04 18:36:32.172703 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:36:32.172534    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc23c596-3a67-46b6-8d23-ab81151e0c5a-tls-certs\") pod \"dc23c596-3a67-46b6-8d23-ab81151e0c5a\" (UID: \"dc23c596-3a67-46b6-8d23-ab81151e0c5a\") "
Aug 04 18:36:32.172703 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:36:32.172564    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc23c596-3a67-46b6-8d23-ab81151e0c5a-dshm\") pod \"dc23c596-3a67-46b6-8d23-ab81151e0c5a\" (UID: \"dc23c596-3a67-46b6-8d23-ab81151e0c5a\") "
Aug 04 18:36:32.172703 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:36:32.172611    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7ncfh\" (UniqueName: \"kubernetes.io/projected/dc23c596-3a67-46b6-8d23-ab81151e0c5a-kube-api-access-7ncfh\") pod \"dc23c596-3a67-46b6-8d23-ab81151e0c5a\" (UID: \"dc23c596-3a67-46b6-8d23-ab81151e0c5a\") "
Aug 04 18:36:32.172703 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:36:32.172638    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc23c596-3a67-46b6-8d23-ab81151e0c5a-tmp-dir\") pod \"dc23c596-3a67-46b6-8d23-ab81151e0c5a\" (UID: \"dc23c596-3a67-46b6-8d23-ab81151e0c5a\") "
Aug 04 18:36:32.172910 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:36:32.172711    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc23c596-3a67-46b6-8d23-ab81151e0c5a-model-cache\") pod \"dc23c596-3a67-46b6-8d23-ab81151e0c5a\" (UID: \"dc23c596-3a67-46b6-8d23-ab81151e0c5a\") "
Aug 04 18:36:32.172910 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:36:32.172840    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc23c596-3a67-46b6-8d23-ab81151e0c5a-home" (OuterVolumeSpecName: "home") pod "dc23c596-3a67-46b6-8d23-ab81151e0c5a" (UID: "dc23c596-3a67-46b6-8d23-ab81151e0c5a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:36:32.173016 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:36:32.172959    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc23c596-3a67-46b6-8d23-ab81151e0c5a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "dc23c596-3a67-46b6-8d23-ab81151e0c5a" (UID: "dc23c596-3a67-46b6-8d23-ab81151e0c5a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:36:32.173069 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:36:32.173030    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc23c596-3a67-46b6-8d23-ab81151e0c5a-home\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:36:32.173069 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:36:32.173048    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc23c596-3a67-46b6-8d23-ab81151e0c5a-model-cache" (OuterVolumeSpecName: "model-cache") pod "dc23c596-3a67-46b6-8d23-ab81151e0c5a" (UID: "dc23c596-3a67-46b6-8d23-ab81151e0c5a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:36:32.174842 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:36:32.174820    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc23c596-3a67-46b6-8d23-ab81151e0c5a-dshm" (OuterVolumeSpecName: "dshm") pod "dc23c596-3a67-46b6-8d23-ab81151e0c5a" (UID: "dc23c596-3a67-46b6-8d23-ab81151e0c5a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:36:32.174842 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:36:32.174827    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc23c596-3a67-46b6-8d23-ab81151e0c5a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "dc23c596-3a67-46b6-8d23-ab81151e0c5a" (UID: "dc23c596-3a67-46b6-8d23-ab81151e0c5a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:36:32.174976 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:36:32.174845    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc23c596-3a67-46b6-8d23-ab81151e0c5a-kube-api-access-7ncfh" (OuterVolumeSpecName: "kube-api-access-7ncfh") pod "dc23c596-3a67-46b6-8d23-ab81151e0c5a" (UID: "dc23c596-3a67-46b6-8d23-ab81151e0c5a"). InnerVolumeSpecName "kube-api-access-7ncfh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:36:32.273816 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:36:32.273729    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7ncfh\" (UniqueName: \"kubernetes.io/projected/dc23c596-3a67-46b6-8d23-ab81151e0c5a-kube-api-access-7ncfh\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:36:32.273816 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:36:32.273761    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc23c596-3a67-46b6-8d23-ab81151e0c5a-tmp-dir\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:36:32.273816 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:36:32.273777    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc23c596-3a67-46b6-8d23-ab81151e0c5a-model-cache\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:36:32.273816 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:36:32.273789    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc23c596-3a67-46b6-8d23-ab81151e0c5a-tls-certs\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:36:32.273816 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:36:32.273802    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc23c596-3a67-46b6-8d23-ab81151e0c5a-dshm\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:36:32.436966 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:36:32.436931    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-bz7tg"]
Aug 04 18:36:32.441172 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:36:32.441139    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-bz7tg"]
Aug 04 18:36:32.831390 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:36:32.831350    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-brl65"
Aug 04 18:36:32.942258 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:36:32.942213    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dc23c596-3a67-46b6-8d23-ab81151e0c5a" path="/var/lib/kubelet/pods/dc23c596-3a67-46b6-8d23-ab81151e0c5a/volumes"
Aug 04 18:36:41.830700 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:36:41.830665    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-brl65"
Aug 04 18:36:41.843413 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:36:41.843385    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-brl65"
Aug 04 18:36:42.183120 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:36:42.183095    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-brl65"
Aug 04 18:36:54.219505 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:36:54.219452    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche9lsr7"]
Aug 04 18:36:54.219910 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:36:54.219895    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc23c596-3a67-46b6-8d23-ab81151e0c5a" containerName="main"
Aug 04 18:36:54.219910 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:36:54.219908    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc23c596-3a67-46b6-8d23-ab81151e0c5a" containerName="main"
Aug 04 18:36:54.220005 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:36:54.219993    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="dc23c596-3a67-46b6-8d23-ab81151e0c5a" containerName="main"
Aug 04 18:36:54.223095 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:36:54.223074    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche9lsr7"
Aug 04 18:36:54.227258 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:36:54.227224    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"kube-root-ca.crt\""
Aug 04 18:36:54.227494 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:36:54.227261    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"openshift-service-ca.crt\""
Aug 04 18:36:54.227494 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:36:54.227224    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"default-dockercfg-7zpwf\""
Aug 04 18:36:54.227841 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:36:54.227809    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-epp-sa-dockercfg-tbbcr\""
Aug 04 18:36:54.228052 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:36:54.228035    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisv08544b88a8d963ffd553cc1f3ed82d16-kserve-self-signed-certs\""
Aug 04 18:36:54.231434 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:36:54.231406    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche9lsr7"]
Aug 04 18:36:54.258633 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:36:54.258594    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef946424-7860-4069-a7fe-a2ddc2da2804-tls-certs\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche9lsr7\" (UID: \"ef946424-7860-4069-a7fe-a2ddc2da2804\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche9lsr7"
Aug 04 18:36:54.258820 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:36:54.258644    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5t26\" (UniqueName: \"kubernetes.io/projected/ef946424-7860-4069-a7fe-a2ddc2da2804-kube-api-access-l5t26\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche9lsr7\" (UID: \"ef946424-7860-4069-a7fe-a2ddc2da2804\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche9lsr7"
Aug 04 18:36:54.359375 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:36:54.359335    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5t26\" (UniqueName: \"kubernetes.io/projected/ef946424-7860-4069-a7fe-a2ddc2da2804-kube-api-access-l5t26\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche9lsr7\" (UID: \"ef946424-7860-4069-a7fe-a2ddc2da2804\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche9lsr7"
Aug 04 18:36:54.359573 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:36:54.359444    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef946424-7860-4069-a7fe-a2ddc2da2804-tls-certs\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche9lsr7\" (UID: \"ef946424-7860-4069-a7fe-a2ddc2da2804\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche9lsr7"
Aug 04 18:36:54.362132 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:36:54.362104    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef946424-7860-4069-a7fe-a2ddc2da2804-tls-certs\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche9lsr7\" (UID: \"ef946424-7860-4069-a7fe-a2ddc2da2804\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche9lsr7"
Aug 04 18:36:54.367970 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:36:54.367944    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5t26\" (UniqueName: \"kubernetes.io/projected/ef946424-7860-4069-a7fe-a2ddc2da2804-kube-api-access-l5t26\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche9lsr7\" (UID: \"ef946424-7860-4069-a7fe-a2ddc2da2804\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche9lsr7"
Aug 04 18:36:54.534333 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:36:54.534241    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche9lsr7"
Aug 04 18:36:54.675096 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:36:54.675071    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche9lsr7"]
Aug 04 18:36:54.676949 ip-10-0-135-124 kubenswrapper[2583]: W0804 18:36:54.676918    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef946424_7860_4069_a7fe_a2ddc2da2804.slice/crio-bf5ae0b5bc97c122a8817e4d49d14ab1bd55883f1da049d80a9fcd6d21e25700 WatchSource:0}: Error finding container bf5ae0b5bc97c122a8817e4d49d14ab1bd55883f1da049d80a9fcd6d21e25700: Status 404 returned error can't find the container with id bf5ae0b5bc97c122a8817e4d49d14ab1bd55883f1da049d80a9fcd6d21e25700
Aug 04 18:36:55.247078 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:36:55.247039    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche9lsr7" event={"ID":"ef946424-7860-4069-a7fe-a2ddc2da2804","Type":"ContainerStarted","Data":"74fa83bbc0a6ec2eae8d6dfd16222f1bbc94e0091cba2d10a1db61efc4a893a3"}
Aug 04 18:36:55.247571 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:36:55.247094    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche9lsr7" event={"ID":"ef946424-7860-4069-a7fe-a2ddc2da2804","Type":"ContainerStarted","Data":"bf5ae0b5bc97c122a8817e4d49d14ab1bd55883f1da049d80a9fcd6d21e25700"}
Aug 04 18:36:55.247571 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:36:55.247183    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche9lsr7"
Aug 04 18:36:55.267987 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:36:55.267926    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche9lsr7" podStartSLOduration=1.267910585 podStartE2EDuration="1.267910585s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:36:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:36:55.26581195 +0000 UTC m=+2228.847030540" watchObservedRunningTime="2026-08-04 18:36:55.267910585 +0000 UTC m=+2228.849129162"
Aug 04 18:37:26.255424 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:37:26.255389    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche9lsr7"
Aug 04 18:37:29.363543 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:37:29.363501    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche9lsr7"]
Aug 04 18:37:36.255314 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:37:36.255268    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche9lsr7" podUID="ef946424-7860-4069-a7fe-a2ddc2da2804" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Aug 04 18:37:44.387056 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:37:44.386996    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche9lsr7" podUID="ef946424-7860-4069-a7fe-a2ddc2da2804" containerName="main" containerID="cri-o://74fa83bbc0a6ec2eae8d6dfd16222f1bbc94e0091cba2d10a1db61efc4a893a3" gracePeriod=45
Aug 04 18:37:44.513178 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:37:44.513142    2583 generic.go:356] "Generic (PLEG): container finished" podID="ef946424-7860-4069-a7fe-a2ddc2da2804" containerID="74fa83bbc0a6ec2eae8d6dfd16222f1bbc94e0091cba2d10a1db61efc4a893a3" exitCode=0
Aug 04 18:37:44.513312 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:37:44.513200    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche9lsr7" event={"ID":"ef946424-7860-4069-a7fe-a2ddc2da2804","Type":"ContainerDied","Data":"74fa83bbc0a6ec2eae8d6dfd16222f1bbc94e0091cba2d10a1db61efc4a893a3"}
Aug 04 18:37:44.635482 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:37:44.635442    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche9lsr7"
Aug 04 18:37:44.700926 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:37:44.700883    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef946424-7860-4069-a7fe-a2ddc2da2804-tls-certs\") pod \"ef946424-7860-4069-a7fe-a2ddc2da2804\" (UID: \"ef946424-7860-4069-a7fe-a2ddc2da2804\") "
Aug 04 18:37:44.701141 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:37:44.700965    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l5t26\" (UniqueName: \"kubernetes.io/projected/ef946424-7860-4069-a7fe-a2ddc2da2804-kube-api-access-l5t26\") pod \"ef946424-7860-4069-a7fe-a2ddc2da2804\" (UID: \"ef946424-7860-4069-a7fe-a2ddc2da2804\") "
Aug 04 18:37:44.703199 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:37:44.703170    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef946424-7860-4069-a7fe-a2ddc2da2804-kube-api-access-l5t26" (OuterVolumeSpecName: "kube-api-access-l5t26") pod "ef946424-7860-4069-a7fe-a2ddc2da2804" (UID: "ef946424-7860-4069-a7fe-a2ddc2da2804"). InnerVolumeSpecName "kube-api-access-l5t26". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:37:44.703343 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:37:44.703197    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef946424-7860-4069-a7fe-a2ddc2da2804-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ef946424-7860-4069-a7fe-a2ddc2da2804" (UID: "ef946424-7860-4069-a7fe-a2ddc2da2804"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:37:44.801871 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:37:44.801839    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef946424-7860-4069-a7fe-a2ddc2da2804-tls-certs\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:37:44.801871 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:37:44.801865    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l5t26\" (UniqueName: \"kubernetes.io/projected/ef946424-7860-4069-a7fe-a2ddc2da2804-kube-api-access-l5t26\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:37:45.519396 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:37:45.519363    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche9lsr7" event={"ID":"ef946424-7860-4069-a7fe-a2ddc2da2804","Type":"ContainerDied","Data":"bf5ae0b5bc97c122a8817e4d49d14ab1bd55883f1da049d80a9fcd6d21e25700"}
Aug 04 18:37:45.519817 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:37:45.519427    2583 scope.go:117] "RemoveContainer" containerID="74fa83bbc0a6ec2eae8d6dfd16222f1bbc94e0091cba2d10a1db61efc4a893a3"
Aug 04 18:37:45.519817 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:37:45.519375    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche9lsr7"
Aug 04 18:37:45.537630 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:37:45.537601    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche9lsr7"]
Aug 04 18:37:45.541200 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:37:45.541179    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche9lsr7"]
Aug 04 18:37:46.943976 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:37:46.943941    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ef946424-7860-4069-a7fe-a2ddc2da2804" path="/var/lib/kubelet/pods/ef946424-7860-4069-a7fe-a2ddc2da2804/volumes"
Aug 04 18:38:04.115000 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:38:04.114962    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-brl65"]
Aug 04 18:38:19.133953 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:38:19.133883    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-brl65" podUID="783c6495-c0e8-4cd3-8dae-b7bbca43c8e8" containerName="main" containerID="cri-o://7294e4a1398960ac3202d771e535fa4fa451b1bec4211300707bcfaeb06cbbc4" gracePeriod=45
Aug 04 18:38:19.172656 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:38:19.172624    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-brl65" podUID="783c6495-c0e8-4cd3-8dae-b7bbca43c8e8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/health\": dial tcp 10.132.0.74:8000: connect: connection refused"
Aug 04 18:38:19.390510 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:38:19.390441    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-brl65"
Aug 04 18:38:19.498387 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:38:19.498355    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6mb8c\" (UniqueName: \"kubernetes.io/projected/783c6495-c0e8-4cd3-8dae-b7bbca43c8e8-kube-api-access-6mb8c\") pod \"783c6495-c0e8-4cd3-8dae-b7bbca43c8e8\" (UID: \"783c6495-c0e8-4cd3-8dae-b7bbca43c8e8\") "
Aug 04 18:38:19.498588 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:38:19.498394    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/783c6495-c0e8-4cd3-8dae-b7bbca43c8e8-tls-certs\") pod \"783c6495-c0e8-4cd3-8dae-b7bbca43c8e8\" (UID: \"783c6495-c0e8-4cd3-8dae-b7bbca43c8e8\") "
Aug 04 18:38:19.498588 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:38:19.498456    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/783c6495-c0e8-4cd3-8dae-b7bbca43c8e8-model-cache\") pod \"783c6495-c0e8-4cd3-8dae-b7bbca43c8e8\" (UID: \"783c6495-c0e8-4cd3-8dae-b7bbca43c8e8\") "
Aug 04 18:38:19.498588 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:38:19.498527    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/783c6495-c0e8-4cd3-8dae-b7bbca43c8e8-tmp-dir\") pod \"783c6495-c0e8-4cd3-8dae-b7bbca43c8e8\" (UID: \"783c6495-c0e8-4cd3-8dae-b7bbca43c8e8\") "
Aug 04 18:38:19.498588 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:38:19.498543    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/783c6495-c0e8-4cd3-8dae-b7bbca43c8e8-home\") pod \"783c6495-c0e8-4cd3-8dae-b7bbca43c8e8\" (UID: \"783c6495-c0e8-4cd3-8dae-b7bbca43c8e8\") "
Aug 04 18:38:19.498588 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:38:19.498566    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/783c6495-c0e8-4cd3-8dae-b7bbca43c8e8-dshm\") pod \"783c6495-c0e8-4cd3-8dae-b7bbca43c8e8\" (UID: \"783c6495-c0e8-4cd3-8dae-b7bbca43c8e8\") "
Aug 04 18:38:19.498879 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:38:19.498787    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/783c6495-c0e8-4cd3-8dae-b7bbca43c8e8-model-cache" (OuterVolumeSpecName: "model-cache") pod "783c6495-c0e8-4cd3-8dae-b7bbca43c8e8" (UID: "783c6495-c0e8-4cd3-8dae-b7bbca43c8e8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:38:19.498879 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:38:19.498803    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/783c6495-c0e8-4cd3-8dae-b7bbca43c8e8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "783c6495-c0e8-4cd3-8dae-b7bbca43c8e8" (UID: "783c6495-c0e8-4cd3-8dae-b7bbca43c8e8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:38:19.498879 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:38:19.498815    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/783c6495-c0e8-4cd3-8dae-b7bbca43c8e8-home" (OuterVolumeSpecName: "home") pod "783c6495-c0e8-4cd3-8dae-b7bbca43c8e8" (UID: "783c6495-c0e8-4cd3-8dae-b7bbca43c8e8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:38:19.500805 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:38:19.500775    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/783c6495-c0e8-4cd3-8dae-b7bbca43c8e8-kube-api-access-6mb8c" (OuterVolumeSpecName: "kube-api-access-6mb8c") pod "783c6495-c0e8-4cd3-8dae-b7bbca43c8e8" (UID: "783c6495-c0e8-4cd3-8dae-b7bbca43c8e8"). InnerVolumeSpecName "kube-api-access-6mb8c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:38:19.500805 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:38:19.500799    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/783c6495-c0e8-4cd3-8dae-b7bbca43c8e8-dshm" (OuterVolumeSpecName: "dshm") pod "783c6495-c0e8-4cd3-8dae-b7bbca43c8e8" (UID: "783c6495-c0e8-4cd3-8dae-b7bbca43c8e8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:38:19.500947 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:38:19.500799    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/783c6495-c0e8-4cd3-8dae-b7bbca43c8e8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "783c6495-c0e8-4cd3-8dae-b7bbca43c8e8" (UID: "783c6495-c0e8-4cd3-8dae-b7bbca43c8e8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:38:19.600202 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:38:19.600161    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/783c6495-c0e8-4cd3-8dae-b7bbca43c8e8-model-cache\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:38:19.600340 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:38:19.600219    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/783c6495-c0e8-4cd3-8dae-b7bbca43c8e8-tmp-dir\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:38:19.600340 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:38:19.600232    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/783c6495-c0e8-4cd3-8dae-b7bbca43c8e8-home\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:38:19.600340 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:38:19.600244    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/783c6495-c0e8-4cd3-8dae-b7bbca43c8e8-dshm\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:38:19.600340 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:38:19.600256    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6mb8c\" (UniqueName: \"kubernetes.io/projected/783c6495-c0e8-4cd3-8dae-b7bbca43c8e8-kube-api-access-6mb8c\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:38:19.600340 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:38:19.600270    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/783c6495-c0e8-4cd3-8dae-b7bbca43c8e8-tls-certs\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:38:19.700371 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:38:19.700337    2583 generic.go:356] "Generic (PLEG): container finished" podID="783c6495-c0e8-4cd3-8dae-b7bbca43c8e8" containerID="7294e4a1398960ac3202d771e535fa4fa451b1bec4211300707bcfaeb06cbbc4" exitCode=0
Aug 04 18:38:19.700558 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:38:19.700399    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-brl65"
Aug 04 18:38:19.700558 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:38:19.700445    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-brl65" event={"ID":"783c6495-c0e8-4cd3-8dae-b7bbca43c8e8","Type":"ContainerDied","Data":"7294e4a1398960ac3202d771e535fa4fa451b1bec4211300707bcfaeb06cbbc4"}
Aug 04 18:38:19.700558 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:38:19.700550    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-brl65" event={"ID":"783c6495-c0e8-4cd3-8dae-b7bbca43c8e8","Type":"ContainerDied","Data":"68b436b75ec0f708adb46a6baa03a3219e06740f4acd00f9b46f013417fcf79f"}
Aug 04 18:38:19.700731 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:38:19.700575    2583 scope.go:117] "RemoveContainer" containerID="7294e4a1398960ac3202d771e535fa4fa451b1bec4211300707bcfaeb06cbbc4"
Aug 04 18:38:19.710715 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:38:19.710695    2583 scope.go:117] "RemoveContainer" containerID="7294e4a1398960ac3202d771e535fa4fa451b1bec4211300707bcfaeb06cbbc4"
Aug 04 18:38:19.710996 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:38:19.710976    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7294e4a1398960ac3202d771e535fa4fa451b1bec4211300707bcfaeb06cbbc4\": container with ID starting with 7294e4a1398960ac3202d771e535fa4fa451b1bec4211300707bcfaeb06cbbc4 not found: ID does not exist" containerID="7294e4a1398960ac3202d771e535fa4fa451b1bec4211300707bcfaeb06cbbc4"
Aug 04 18:38:19.711056 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:38:19.711006    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7294e4a1398960ac3202d771e535fa4fa451b1bec4211300707bcfaeb06cbbc4"} err="failed to get container status \"7294e4a1398960ac3202d771e535fa4fa451b1bec4211300707bcfaeb06cbbc4\": rpc error: code = NotFound desc = could not find container \"7294e4a1398960ac3202d771e535fa4fa451b1bec4211300707bcfaeb06cbbc4\": container with ID starting with 7294e4a1398960ac3202d771e535fa4fa451b1bec4211300707bcfaeb06cbbc4 not found: ID does not exist"
Aug 04 18:38:19.725699 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:38:19.725674    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-brl65"]
Aug 04 18:38:19.729080 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:38:19.729060    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-brl65"]
Aug 04 18:38:20.941846 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:38:20.941812    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="783c6495-c0e8-4cd3-8dae-b7bbca43c8e8" path="/var/lib/kubelet/pods/783c6495-c0e8-4cd3-8dae-b7bbca43c8e8/volumes"
Aug 04 18:38:31.708051 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:38:31.708017    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-5ckmn"]
Aug 04 18:38:31.708459 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:38:31.708366    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef946424-7860-4069-a7fe-a2ddc2da2804" containerName="main"
Aug 04 18:38:31.708459 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:38:31.708376    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef946424-7860-4069-a7fe-a2ddc2da2804" containerName="main"
Aug 04 18:38:31.708459 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:38:31.708393    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="783c6495-c0e8-4cd3-8dae-b7bbca43c8e8" containerName="main"
Aug 04 18:38:31.708459 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:38:31.708399    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="783c6495-c0e8-4cd3-8dae-b7bbca43c8e8" containerName="main"
Aug 04 18:38:31.708459 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:38:31.708459    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef946424-7860-4069-a7fe-a2ddc2da2804" containerName="main"
Aug 04 18:38:31.708459 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:38:31.708485    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="783c6495-c0e8-4cd3-8dae-b7bbca43c8e8" containerName="main"
Aug 04 18:38:31.711500 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:38:31.711458    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-5ckmn"
Aug 04 18:38:31.714703 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:38:31.714682    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-delete-at-nonzero-weight-6b4017d2\"/\"kube-root-ca.crt\""
Aug 04 18:38:31.715773 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:38:31.715753    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-delete-at-nonzero-weight-6b4017d2\"/\"openshift-service-ca.crt\""
Aug 04 18:38:31.715906 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:38:31.715752    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-delete-at-nonzero-weight-6b4017d2\"/\"default-dockercfg-gll56\""
Aug 04 18:38:31.715906 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:38:31.715898    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-delete-at-nonzero-weight-6b4017d2\"/\"delw-v2-kserve-self-signed-certs\""
Aug 04 18:38:31.719877 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:38:31.719856    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-5ckmn"]
Aug 04 18:38:31.894261 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:38:31.894220    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99c1175e-4f66-47fc-acf4-3939ed4223f0-dshm\") pod \"delw-v2-kserve-5b84cc7ddf-5ckmn\" (UID: \"99c1175e-4f66-47fc-acf4-3939ed4223f0\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-5ckmn"
Aug 04 18:38:31.894424 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:38:31.894279    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99c1175e-4f66-47fc-acf4-3939ed4223f0-model-cache\") pod \"delw-v2-kserve-5b84cc7ddf-5ckmn\" (UID: \"99c1175e-4f66-47fc-acf4-3939ed4223f0\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-5ckmn"
Aug 04 18:38:31.894424 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:38:31.894345    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkrx2\" (UniqueName: \"kubernetes.io/projected/99c1175e-4f66-47fc-acf4-3939ed4223f0-kube-api-access-kkrx2\") pod \"delw-v2-kserve-5b84cc7ddf-5ckmn\" (UID: \"99c1175e-4f66-47fc-acf4-3939ed4223f0\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-5ckmn"
Aug 04 18:38:31.894424 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:38:31.894409    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99c1175e-4f66-47fc-acf4-3939ed4223f0-home\") pod \"delw-v2-kserve-5b84cc7ddf-5ckmn\" (UID: \"99c1175e-4f66-47fc-acf4-3939ed4223f0\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-5ckmn"
Aug 04 18:38:31.894555 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:38:31.894432    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99c1175e-4f66-47fc-acf4-3939ed4223f0-tls-certs\") pod \"delw-v2-kserve-5b84cc7ddf-5ckmn\" (UID: \"99c1175e-4f66-47fc-acf4-3939ed4223f0\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-5ckmn"
Aug 04 18:38:31.894555 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:38:31.894456    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99c1175e-4f66-47fc-acf4-3939ed4223f0-tmp-dir\") pod \"delw-v2-kserve-5b84cc7ddf-5ckmn\" (UID: \"99c1175e-4f66-47fc-acf4-3939ed4223f0\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-5ckmn"
Aug 04 18:38:31.995867 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:38:31.995791    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99c1175e-4f66-47fc-acf4-3939ed4223f0-home\") pod \"delw-v2-kserve-5b84cc7ddf-5ckmn\" (UID: \"99c1175e-4f66-47fc-acf4-3939ed4223f0\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-5ckmn"
Aug 04 18:38:31.995867 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:38:31.995829    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99c1175e-4f66-47fc-acf4-3939ed4223f0-tls-certs\") pod \"delw-v2-kserve-5b84cc7ddf-5ckmn\" (UID: \"99c1175e-4f66-47fc-acf4-3939ed4223f0\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-5ckmn"
Aug 04 18:38:31.995867 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:38:31.995850    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99c1175e-4f66-47fc-acf4-3939ed4223f0-tmp-dir\") pod \"delw-v2-kserve-5b84cc7ddf-5ckmn\" (UID: \"99c1175e-4f66-47fc-acf4-3939ed4223f0\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-5ckmn"
Aug 04 18:38:31.996080 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:38:31.995905    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99c1175e-4f66-47fc-acf4-3939ed4223f0-dshm\") pod \"delw-v2-kserve-5b84cc7ddf-5ckmn\" (UID: \"99c1175e-4f66-47fc-acf4-3939ed4223f0\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-5ckmn"
Aug 04 18:38:31.996080 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:38:31.995953    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99c1175e-4f66-47fc-acf4-3939ed4223f0-model-cache\") pod \"delw-v2-kserve-5b84cc7ddf-5ckmn\" (UID: \"99c1175e-4f66-47fc-acf4-3939ed4223f0\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-5ckmn"
Aug 04 18:38:31.996080 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:38:31.995987    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkrx2\" (UniqueName: \"kubernetes.io/projected/99c1175e-4f66-47fc-acf4-3939ed4223f0-kube-api-access-kkrx2\") pod \"delw-v2-kserve-5b84cc7ddf-5ckmn\" (UID: \"99c1175e-4f66-47fc-acf4-3939ed4223f0\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-5ckmn"
Aug 04 18:38:31.996250 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:38:31.996231    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99c1175e-4f66-47fc-acf4-3939ed4223f0-home\") pod \"delw-v2-kserve-5b84cc7ddf-5ckmn\" (UID: \"99c1175e-4f66-47fc-acf4-3939ed4223f0\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-5ckmn"
Aug 04 18:38:31.996324 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:38:31.996298    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99c1175e-4f66-47fc-acf4-3939ed4223f0-tmp-dir\") pod \"delw-v2-kserve-5b84cc7ddf-5ckmn\" (UID: \"99c1175e-4f66-47fc-acf4-3939ed4223f0\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-5ckmn"
Aug 04 18:38:31.996400 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:38:31.996376    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99c1175e-4f66-47fc-acf4-3939ed4223f0-model-cache\") pod \"delw-v2-kserve-5b84cc7ddf-5ckmn\" (UID: \"99c1175e-4f66-47fc-acf4-3939ed4223f0\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-5ckmn"
Aug 04 18:38:31.998308 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:38:31.998284    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99c1175e-4f66-47fc-acf4-3939ed4223f0-dshm\") pod \"delw-v2-kserve-5b84cc7ddf-5ckmn\" (UID: \"99c1175e-4f66-47fc-acf4-3939ed4223f0\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-5ckmn"
Aug 04 18:38:31.998441 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:38:31.998425    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99c1175e-4f66-47fc-acf4-3939ed4223f0-tls-certs\") pod \"delw-v2-kserve-5b84cc7ddf-5ckmn\" (UID: \"99c1175e-4f66-47fc-acf4-3939ed4223f0\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-5ckmn"
Aug 04 18:38:32.004312 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:38:32.004287    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkrx2\" (UniqueName: \"kubernetes.io/projected/99c1175e-4f66-47fc-acf4-3939ed4223f0-kube-api-access-kkrx2\") pod \"delw-v2-kserve-5b84cc7ddf-5ckmn\" (UID: \"99c1175e-4f66-47fc-acf4-3939ed4223f0\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-5ckmn"
Aug 04 18:38:32.023867 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:38:32.023845    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-5ckmn"
Aug 04 18:38:32.169068 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:38:32.169039    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-5ckmn"]
Aug 04 18:38:32.170712 ip-10-0-135-124 kubenswrapper[2583]: W0804 18:38:32.170681    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99c1175e_4f66_47fc_acf4_3939ed4223f0.slice/crio-54e7d74167910609f2f21ead83276f9692279777706aaa2a33693d5a6fc9552b WatchSource:0}: Error finding container 54e7d74167910609f2f21ead83276f9692279777706aaa2a33693d5a6fc9552b: Status 404 returned error can't find the container with id 54e7d74167910609f2f21ead83276f9692279777706aaa2a33693d5a6fc9552b
Aug 04 18:38:32.769926 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:38:32.769889    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-5ckmn" event={"ID":"99c1175e-4f66-47fc-acf4-3939ed4223f0","Type":"ContainerStarted","Data":"54ebf41903638049f2645204c747ba165fa4186900ead9dcf1105b1f586c976a"}
Aug 04 18:38:32.770305 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:38:32.769950    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-5ckmn" event={"ID":"99c1175e-4f66-47fc-acf4-3939ed4223f0","Type":"ContainerStarted","Data":"54e7d74167910609f2f21ead83276f9692279777706aaa2a33693d5a6fc9552b"}
Aug 04 18:38:32.790518 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:38:32.790453    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-5ckmn" podStartSLOduration=1.7904381539999998 podStartE2EDuration="1.790438154s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:38:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:38:32.788498388 +0000 UTC m=+2326.369716962" watchObservedRunningTime="2026-08-04 18:38:32.790438154 +0000 UTC m=+2326.371656732"
Aug 04 18:38:33.024480 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:38:33.024386    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-5ckmn"
Aug 04 18:38:42.024635 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:38:42.024601    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-5ckmn"
Aug 04 18:38:42.037293 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:38:42.037264    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-5ckmn"
Aug 04 18:38:42.831487 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:38:42.831443    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-5ckmn"
Aug 04 18:38:43.602862 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:38:43.602826    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-66c57bb459-ms2z7"]
Aug 04 18:38:43.606517 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:38:43.606495    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-66c57bb459-ms2z7"
Aug 04 18:38:43.609587 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:38:43.609564    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"seaweedfs-s3-creds\""
Aug 04 18:38:43.609705 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:38:43.609566    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"odh-kserve-custom-ca-bundle\""
Aug 04 18:38:43.609705 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:38:43.609599    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"kube-root-ca.crt\""
Aug 04 18:38:43.609705 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:38:43.609598    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"openshift-service-ca.crt\""
Aug 04 18:38:43.609885 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:38:43.609707    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"llmisv3e414c2ba058a022dfd694dbcbac5b51-kserve-self-signed-certs\""
Aug 04 18:38:43.609885 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:38:43.609855    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"default-dockercfg-2s5gb\""
Aug 04 18:38:43.619083 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:38:43.619060    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-66c57bb459-ms2z7"]
Aug 04 18:38:43.689991 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:38:43.689964    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/81d123ff-92f3-40a2-a782-2240a40bf0a2-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-66c57bb459-ms2z7\" (UID: \"81d123ff-92f3-40a2-a782-2240a40bf0a2\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-66c57bb459-ms2z7"
Aug 04 18:38:43.690121 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:38:43.690021    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81d123ff-92f3-40a2-a782-2240a40bf0a2-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-66c57bb459-ms2z7\" (UID: \"81d123ff-92f3-40a2-a782-2240a40bf0a2\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-66c57bb459-ms2z7"
Aug 04 18:38:43.690121 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:38:43.690042    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81d123ff-92f3-40a2-a782-2240a40bf0a2-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-66c57bb459-ms2z7\" (UID: \"81d123ff-92f3-40a2-a782-2240a40bf0a2\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-66c57bb459-ms2z7"
Aug 04 18:38:43.690121 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:38:43.690066    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5j2zh\" (UniqueName: \"kubernetes.io/projected/81d123ff-92f3-40a2-a782-2240a40bf0a2-kube-api-access-5j2zh\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-66c57bb459-ms2z7\" (UID: \"81d123ff-92f3-40a2-a782-2240a40bf0a2\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-66c57bb459-ms2z7"
Aug 04 18:38:43.690121 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:38:43.690093    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81d123ff-92f3-40a2-a782-2240a40bf0a2-home\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-66c57bb459-ms2z7\" (UID: \"81d123ff-92f3-40a2-a782-2240a40bf0a2\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-66c57bb459-ms2z7"
Aug 04 18:38:43.690263 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:38:43.690138    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81d123ff-92f3-40a2-a782-2240a40bf0a2-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-66c57bb459-ms2z7\" (UID: \"81d123ff-92f3-40a2-a782-2240a40bf0a2\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-66c57bb459-ms2z7"
Aug 04 18:38:43.690263 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:38:43.690162    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81d123ff-92f3-40a2-a782-2240a40bf0a2-dshm\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-66c57bb459-ms2z7\" (UID: \"81d123ff-92f3-40a2-a782-2240a40bf0a2\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-66c57bb459-ms2z7"
Aug 04 18:38:43.690263 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:38:43.690186    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/81d123ff-92f3-40a2-a782-2240a40bf0a2-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-66c57bb459-ms2z7\" (UID: \"81d123ff-92f3-40a2-a782-2240a40bf0a2\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-66c57bb459-ms2z7"
Aug 04 18:38:43.790678 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:38:43.790652    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5j2zh\" (UniqueName: \"kubernetes.io/projected/81d123ff-92f3-40a2-a782-2240a40bf0a2-kube-api-access-5j2zh\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-66c57bb459-ms2z7\" (UID: \"81d123ff-92f3-40a2-a782-2240a40bf0a2\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-66c57bb459-ms2z7"
Aug 04 18:38:43.790829 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:38:43.790693    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81d123ff-92f3-40a2-a782-2240a40bf0a2-home\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-66c57bb459-ms2z7\" (UID: \"81d123ff-92f3-40a2-a782-2240a40bf0a2\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-66c57bb459-ms2z7"
Aug 04 18:38:43.790829 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:38:43.790714    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81d123ff-92f3-40a2-a782-2240a40bf0a2-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-66c57bb459-ms2z7\" (UID: \"81d123ff-92f3-40a2-a782-2240a40bf0a2\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-66c57bb459-ms2z7"
Aug 04 18:38:43.790829 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:38:43.790733    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81d123ff-92f3-40a2-a782-2240a40bf0a2-dshm\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-66c57bb459-ms2z7\" (UID: \"81d123ff-92f3-40a2-a782-2240a40bf0a2\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-66c57bb459-ms2z7"
Aug 04 18:38:43.790829 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:38:43.790752    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/81d123ff-92f3-40a2-a782-2240a40bf0a2-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-66c57bb459-ms2z7\" (UID: \"81d123ff-92f3-40a2-a782-2240a40bf0a2\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-66c57bb459-ms2z7"
Aug 04 18:38:43.790829 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:38:43.790781    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/81d123ff-92f3-40a2-a782-2240a40bf0a2-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-66c57bb459-ms2z7\" (UID: \"81d123ff-92f3-40a2-a782-2240a40bf0a2\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-66c57bb459-ms2z7"
Aug 04 18:38:43.791168 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:38:43.790859    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81d123ff-92f3-40a2-a782-2240a40bf0a2-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-66c57bb459-ms2z7\" (UID: \"81d123ff-92f3-40a2-a782-2240a40bf0a2\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-66c57bb459-ms2z7"
Aug 04 18:38:43.791168 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:38:43.791007    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81d123ff-92f3-40a2-a782-2240a40bf0a2-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-66c57bb459-ms2z7\" (UID: \"81d123ff-92f3-40a2-a782-2240a40bf0a2\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-66c57bb459-ms2z7"
Aug 04 18:38:43.791263 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:38:43.791164    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81d123ff-92f3-40a2-a782-2240a40bf0a2-home\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-66c57bb459-ms2z7\" (UID: \"81d123ff-92f3-40a2-a782-2240a40bf0a2\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-66c57bb459-ms2z7"
Aug 04 18:38:43.791263 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:38:43.791196    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81d123ff-92f3-40a2-a782-2240a40bf0a2-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-66c57bb459-ms2z7\" (UID: \"81d123ff-92f3-40a2-a782-2240a40bf0a2\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-66c57bb459-ms2z7"
Aug 04 18:38:43.791263 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:38:43.791254    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/81d123ff-92f3-40a2-a782-2240a40bf0a2-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-66c57bb459-ms2z7\" (UID: \"81d123ff-92f3-40a2-a782-2240a40bf0a2\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-66c57bb459-ms2z7"
Aug 04 18:38:43.791363 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:38:43.791325    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81d123ff-92f3-40a2-a782-2240a40bf0a2-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-66c57bb459-ms2z7\" (UID: \"81d123ff-92f3-40a2-a782-2240a40bf0a2\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-66c57bb459-ms2z7"
Aug 04 18:38:43.791587 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:38:43.791570    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/81d123ff-92f3-40a2-a782-2240a40bf0a2-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-66c57bb459-ms2z7\" (UID: \"81d123ff-92f3-40a2-a782-2240a40bf0a2\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-66c57bb459-ms2z7"
Aug 04 18:38:43.793092 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:38:43.793071    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81d123ff-92f3-40a2-a782-2240a40bf0a2-dshm\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-66c57bb459-ms2z7\" (UID: \"81d123ff-92f3-40a2-a782-2240a40bf0a2\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-66c57bb459-ms2z7"
Aug 04 18:38:43.793395 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:38:43.793377    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81d123ff-92f3-40a2-a782-2240a40bf0a2-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-66c57bb459-ms2z7\" (UID: \"81d123ff-92f3-40a2-a782-2240a40bf0a2\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-66c57bb459-ms2z7"
Aug 04 18:38:43.800080 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:38:43.800056    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5j2zh\" (UniqueName: \"kubernetes.io/projected/81d123ff-92f3-40a2-a782-2240a40bf0a2-kube-api-access-5j2zh\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-66c57bb459-ms2z7\" (UID: \"81d123ff-92f3-40a2-a782-2240a40bf0a2\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-66c57bb459-ms2z7"
Aug 04 18:38:43.917652 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:38:43.917584    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-66c57bb459-ms2z7"
Aug 04 18:38:44.055580 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:38:44.055553    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-66c57bb459-ms2z7"]
Aug 04 18:38:44.057179 ip-10-0-135-124 kubenswrapper[2583]: W0804 18:38:44.057154    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81d123ff_92f3_40a2_a782_2240a40bf0a2.slice/crio-6d190c7098034950797ce4b8886945d4ac6b7234b82634d7227b877136e0ac5c WatchSource:0}: Error finding container 6d190c7098034950797ce4b8886945d4ac6b7234b82634d7227b877136e0ac5c: Status 404 returned error can't find the container with id 6d190c7098034950797ce4b8886945d4ac6b7234b82634d7227b877136e0ac5c
Aug 04 18:38:44.834033 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:38:44.833997    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-66c57bb459-ms2z7" event={"ID":"81d123ff-92f3-40a2-a782-2240a40bf0a2","Type":"ContainerStarted","Data":"7040d4179cc66ff86e8ee40bc562985f34c0dcf9aa0a6e2500cbf1dee9928a15"}
Aug 04 18:38:44.834429 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:38:44.834084    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-66c57bb459-ms2z7" event={"ID":"81d123ff-92f3-40a2-a782-2240a40bf0a2","Type":"ContainerStarted","Data":"6d190c7098034950797ce4b8886945d4ac6b7234b82634d7227b877136e0ac5c"}
Aug 04 18:38:47.854951 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:38:47.854916    2583 generic.go:356] "Generic (PLEG): container finished" podID="81d123ff-92f3-40a2-a782-2240a40bf0a2" containerID="7040d4179cc66ff86e8ee40bc562985f34c0dcf9aa0a6e2500cbf1dee9928a15" exitCode=0
Aug 04 18:38:47.855356 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:38:47.854995    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-66c57bb459-ms2z7" event={"ID":"81d123ff-92f3-40a2-a782-2240a40bf0a2","Type":"ContainerDied","Data":"7040d4179cc66ff86e8ee40bc562985f34c0dcf9aa0a6e2500cbf1dee9928a15"}
Aug 04 18:38:48.864315 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:38:48.864281    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-66c57bb459-ms2z7" event={"ID":"81d123ff-92f3-40a2-a782-2240a40bf0a2","Type":"ContainerStarted","Data":"42114d387b7129a6aeda237adc2cb88f201df7bb1375ba98d041f2a9630275f0"}
Aug 04 18:38:48.890014 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:38:48.889966    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-66c57bb459-ms2z7" podStartSLOduration=3.042540099 podStartE2EDuration="5.889950881s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.847410782s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:38:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:38:48.887375683 +0000 UTC m=+2342.468594262" watchObservedRunningTime="2026-08-04 18:38:48.889950881 +0000 UTC m=+2342.471169459"
Aug 04 18:38:48.917866 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:38:48.917831    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-66c57bb459-ms2z7"
Aug 04 18:38:53.918316 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:38:53.918280    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-66c57bb459-ms2z7"
Aug 04 18:38:53.919801 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:38:53.919769    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-66c57bb459-ms2z7" podUID="81d123ff-92f3-40a2-a782-2240a40bf0a2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/health\": dial tcp 10.132.0.77:8000: connect: connection refused"
Aug 04 18:39:03.918298 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:39:03.918264    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-66c57bb459-ms2z7" podUID="81d123ff-92f3-40a2-a782-2240a40bf0a2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/health\": dial tcp 10.132.0.77:8000: connect: connection refused"
Aug 04 18:39:13.918225 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:39:13.918184    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-66c57bb459-ms2z7" podUID="81d123ff-92f3-40a2-a782-2240a40bf0a2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/health\": dial tcp 10.132.0.77:8000: connect: connection refused"
Aug 04 18:39:18.496923 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:39:18.496887    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-5ckmn"]
Aug 04 18:39:23.918313 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:39:23.918275    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-66c57bb459-ms2z7" podUID="81d123ff-92f3-40a2-a782-2240a40bf0a2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/health\": dial tcp 10.132.0.77:8000: connect: connection refused"
Aug 04 18:39:33.515685 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:39:33.515644    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-5ckmn" podUID="99c1175e-4f66-47fc-acf4-3939ed4223f0" containerName="main" containerID="cri-o://54ebf41903638049f2645204c747ba165fa4186900ead9dcf1105b1f586c976a" gracePeriod=45
Aug 04 18:39:33.787931 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:39:33.787906    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-5ckmn"
Aug 04 18:39:33.918740 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:39:33.918699    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-66c57bb459-ms2z7" podUID="81d123ff-92f3-40a2-a782-2240a40bf0a2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/health\": dial tcp 10.132.0.77:8000: connect: connection refused"
Aug 04 18:39:33.941809 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:39:33.941778    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99c1175e-4f66-47fc-acf4-3939ed4223f0-tls-certs\") pod \"99c1175e-4f66-47fc-acf4-3939ed4223f0\" (UID: \"99c1175e-4f66-47fc-acf4-3939ed4223f0\") "
Aug 04 18:39:33.941965 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:39:33.941822    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kkrx2\" (UniqueName: \"kubernetes.io/projected/99c1175e-4f66-47fc-acf4-3939ed4223f0-kube-api-access-kkrx2\") pod \"99c1175e-4f66-47fc-acf4-3939ed4223f0\" (UID: \"99c1175e-4f66-47fc-acf4-3939ed4223f0\") "
Aug 04 18:39:33.941965 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:39:33.941842    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99c1175e-4f66-47fc-acf4-3939ed4223f0-home\") pod \"99c1175e-4f66-47fc-acf4-3939ed4223f0\" (UID: \"99c1175e-4f66-47fc-acf4-3939ed4223f0\") "
Aug 04 18:39:33.941965 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:39:33.941911    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99c1175e-4f66-47fc-acf4-3939ed4223f0-model-cache\") pod \"99c1175e-4f66-47fc-acf4-3939ed4223f0\" (UID: \"99c1175e-4f66-47fc-acf4-3939ed4223f0\") "
Aug 04 18:39:33.942130 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:39:33.941975    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99c1175e-4f66-47fc-acf4-3939ed4223f0-tmp-dir\") pod \"99c1175e-4f66-47fc-acf4-3939ed4223f0\" (UID: \"99c1175e-4f66-47fc-acf4-3939ed4223f0\") "
Aug 04 18:39:33.942130 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:39:33.942007    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99c1175e-4f66-47fc-acf4-3939ed4223f0-dshm\") pod \"99c1175e-4f66-47fc-acf4-3939ed4223f0\" (UID: \"99c1175e-4f66-47fc-acf4-3939ed4223f0\") "
Aug 04 18:39:33.942242 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:39:33.942210    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99c1175e-4f66-47fc-acf4-3939ed4223f0-home" (OuterVolumeSpecName: "home") pod "99c1175e-4f66-47fc-acf4-3939ed4223f0" (UID: "99c1175e-4f66-47fc-acf4-3939ed4223f0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:39:33.942297 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:39:33.942239    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99c1175e-4f66-47fc-acf4-3939ed4223f0-model-cache" (OuterVolumeSpecName: "model-cache") pod "99c1175e-4f66-47fc-acf4-3939ed4223f0" (UID: "99c1175e-4f66-47fc-acf4-3939ed4223f0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:39:33.942429 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:39:33.942401    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99c1175e-4f66-47fc-acf4-3939ed4223f0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "99c1175e-4f66-47fc-acf4-3939ed4223f0" (UID: "99c1175e-4f66-47fc-acf4-3939ed4223f0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:39:33.944259 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:39:33.944237    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99c1175e-4f66-47fc-acf4-3939ed4223f0-dshm" (OuterVolumeSpecName: "dshm") pod "99c1175e-4f66-47fc-acf4-3939ed4223f0" (UID: "99c1175e-4f66-47fc-acf4-3939ed4223f0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:39:33.944259 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:39:33.944247    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99c1175e-4f66-47fc-acf4-3939ed4223f0-kube-api-access-kkrx2" (OuterVolumeSpecName: "kube-api-access-kkrx2") pod "99c1175e-4f66-47fc-acf4-3939ed4223f0" (UID: "99c1175e-4f66-47fc-acf4-3939ed4223f0"). InnerVolumeSpecName "kube-api-access-kkrx2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:39:33.944403 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:39:33.944269    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99c1175e-4f66-47fc-acf4-3939ed4223f0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "99c1175e-4f66-47fc-acf4-3939ed4223f0" (UID: "99c1175e-4f66-47fc-acf4-3939ed4223f0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:39:34.043690 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:39:34.043609    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99c1175e-4f66-47fc-acf4-3939ed4223f0-tmp-dir\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:39:34.043690 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:39:34.043643    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99c1175e-4f66-47fc-acf4-3939ed4223f0-dshm\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:39:34.043690 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:39:34.043655    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99c1175e-4f66-47fc-acf4-3939ed4223f0-tls-certs\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:39:34.043690 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:39:34.043667    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kkrx2\" (UniqueName: \"kubernetes.io/projected/99c1175e-4f66-47fc-acf4-3939ed4223f0-kube-api-access-kkrx2\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:39:34.043690 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:39:34.043679    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99c1175e-4f66-47fc-acf4-3939ed4223f0-home\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:39:34.043690 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:39:34.043691    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99c1175e-4f66-47fc-acf4-3939ed4223f0-model-cache\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:39:34.128003 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:39:34.127967    2583 generic.go:356] "Generic (PLEG): container finished" podID="99c1175e-4f66-47fc-acf4-3939ed4223f0" containerID="54ebf41903638049f2645204c747ba165fa4186900ead9dcf1105b1f586c976a" exitCode=0
Aug 04 18:39:34.128177 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:39:34.128013    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-5ckmn" event={"ID":"99c1175e-4f66-47fc-acf4-3939ed4223f0","Type":"ContainerDied","Data":"54ebf41903638049f2645204c747ba165fa4186900ead9dcf1105b1f586c976a"}
Aug 04 18:39:34.128177 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:39:34.128017    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-5ckmn"
Aug 04 18:39:34.128177 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:39:34.128057    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-5ckmn" event={"ID":"99c1175e-4f66-47fc-acf4-3939ed4223f0","Type":"ContainerDied","Data":"54e7d74167910609f2f21ead83276f9692279777706aaa2a33693d5a6fc9552b"}
Aug 04 18:39:34.128177 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:39:34.128080    2583 scope.go:117] "RemoveContainer" containerID="54ebf41903638049f2645204c747ba165fa4186900ead9dcf1105b1f586c976a"
Aug 04 18:39:34.139562 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:39:34.139540    2583 scope.go:117] "RemoveContainer" containerID="54ebf41903638049f2645204c747ba165fa4186900ead9dcf1105b1f586c976a"
Aug 04 18:39:34.139861 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:39:34.139840    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54ebf41903638049f2645204c747ba165fa4186900ead9dcf1105b1f586c976a\": container with ID starting with 54ebf41903638049f2645204c747ba165fa4186900ead9dcf1105b1f586c976a not found: ID does not exist" containerID="54ebf41903638049f2645204c747ba165fa4186900ead9dcf1105b1f586c976a"
Aug 04 18:39:34.139945 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:39:34.139872    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54ebf41903638049f2645204c747ba165fa4186900ead9dcf1105b1f586c976a"} err="failed to get container status \"54ebf41903638049f2645204c747ba165fa4186900ead9dcf1105b1f586c976a\": rpc error: code = NotFound desc = could not find container \"54ebf41903638049f2645204c747ba165fa4186900ead9dcf1105b1f586c976a\": container with ID starting with 54ebf41903638049f2645204c747ba165fa4186900ead9dcf1105b1f586c976a not found: ID does not exist"
Aug 04 18:39:34.153767 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:39:34.153737    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-5ckmn"]
Aug 04 18:39:34.155834 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:39:34.155811    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-5ckmn"]
Aug 04 18:39:34.942097 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:39:34.942061    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="99c1175e-4f66-47fc-acf4-3939ed4223f0" path="/var/lib/kubelet/pods/99c1175e-4f66-47fc-acf4-3939ed4223f0/volumes"
Aug 04 18:39:43.918493 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:39:43.918433    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-66c57bb459-ms2z7" podUID="81d123ff-92f3-40a2-a782-2240a40bf0a2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/health\": dial tcp 10.132.0.77:8000: connect: connection refused"
Aug 04 18:39:47.105510 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:39:47.105382    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-wskfb_6157c2d3-2fd0-4544-88ad-434ac75f8c04/console-operator/1.log"
Aug 04 18:39:47.114962 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:39:47.114942    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-wskfb_6157c2d3-2fd0-4544-88ad-434ac75f8c04/console-operator/1.log"
Aug 04 18:39:53.918943 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:39:53.918903    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-66c57bb459-ms2z7" podUID="81d123ff-92f3-40a2-a782-2240a40bf0a2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/health\": dial tcp 10.132.0.77:8000: connect: connection refused"
Aug 04 18:40:03.928416 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:40:03.928381    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-66c57bb459-ms2z7"
Aug 04 18:40:03.938425 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:40:03.938392    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-66c57bb459-ms2z7"
Aug 04 18:40:36.677749 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:40:36.677713    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-66c57bb459-ms2z7"]
Aug 04 18:40:51.700041 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:40:51.699969    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-66c57bb459-ms2z7" podUID="81d123ff-92f3-40a2-a782-2240a40bf0a2" containerName="main" containerID="cri-o://42114d387b7129a6aeda237adc2cb88f201df7bb1375ba98d041f2a9630275f0" gracePeriod=45
Aug 04 18:40:51.929735 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:40:51.929687    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-66c57bb459-ms2z7" podUID="81d123ff-92f3-40a2-a782-2240a40bf0a2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/health\": dial tcp 10.132.0.77:8000: connect: connection refused"
Aug 04 18:40:52.353282 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:40:52.353256    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-66c57bb459-ms2z7"
Aug 04 18:40:52.441158 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:40:52.441127    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/81d123ff-92f3-40a2-a782-2240a40bf0a2-kserve-provision-location\") pod \"81d123ff-92f3-40a2-a782-2240a40bf0a2\" (UID: \"81d123ff-92f3-40a2-a782-2240a40bf0a2\") "
Aug 04 18:40:52.441343 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:40:52.441165    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81d123ff-92f3-40a2-a782-2240a40bf0a2-home\") pod \"81d123ff-92f3-40a2-a782-2240a40bf0a2\" (UID: \"81d123ff-92f3-40a2-a782-2240a40bf0a2\") "
Aug 04 18:40:52.441343 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:40:52.441187    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81d123ff-92f3-40a2-a782-2240a40bf0a2-model-cache\") pod \"81d123ff-92f3-40a2-a782-2240a40bf0a2\" (UID: \"81d123ff-92f3-40a2-a782-2240a40bf0a2\") "
Aug 04 18:40:52.441343 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:40:52.441249    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81d123ff-92f3-40a2-a782-2240a40bf0a2-dshm\") pod \"81d123ff-92f3-40a2-a782-2240a40bf0a2\" (UID: \"81d123ff-92f3-40a2-a782-2240a40bf0a2\") "
Aug 04 18:40:52.441343 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:40:52.441271    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/81d123ff-92f3-40a2-a782-2240a40bf0a2-cabundle-cert\") pod \"81d123ff-92f3-40a2-a782-2240a40bf0a2\" (UID: \"81d123ff-92f3-40a2-a782-2240a40bf0a2\") "
Aug 04 18:40:52.441343 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:40:52.441298    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5j2zh\" (UniqueName: \"kubernetes.io/projected/81d123ff-92f3-40a2-a782-2240a40bf0a2-kube-api-access-5j2zh\") pod \"81d123ff-92f3-40a2-a782-2240a40bf0a2\" (UID: \"81d123ff-92f3-40a2-a782-2240a40bf0a2\") "
Aug 04 18:40:52.441650 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:40:52.441363    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81d123ff-92f3-40a2-a782-2240a40bf0a2-tls-certs\") pod \"81d123ff-92f3-40a2-a782-2240a40bf0a2\" (UID: \"81d123ff-92f3-40a2-a782-2240a40bf0a2\") "
Aug 04 18:40:52.441650 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:40:52.441394    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81d123ff-92f3-40a2-a782-2240a40bf0a2-tmp-dir\") pod \"81d123ff-92f3-40a2-a782-2240a40bf0a2\" (UID: \"81d123ff-92f3-40a2-a782-2240a40bf0a2\") "
Aug 04 18:40:52.441650 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:40:52.441582    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81d123ff-92f3-40a2-a782-2240a40bf0a2-model-cache" (OuterVolumeSpecName: "model-cache") pod "81d123ff-92f3-40a2-a782-2240a40bf0a2" (UID: "81d123ff-92f3-40a2-a782-2240a40bf0a2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:40:52.441797 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:40:52.441675    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81d123ff-92f3-40a2-a782-2240a40bf0a2-home" (OuterVolumeSpecName: "home") pod "81d123ff-92f3-40a2-a782-2240a40bf0a2" (UID: "81d123ff-92f3-40a2-a782-2240a40bf0a2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:40:52.441797 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:40:52.441724    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81d123ff-92f3-40a2-a782-2240a40bf0a2-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "81d123ff-92f3-40a2-a782-2240a40bf0a2" (UID: "81d123ff-92f3-40a2-a782-2240a40bf0a2"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:40:52.441907 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:40:52.441806    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81d123ff-92f3-40a2-a782-2240a40bf0a2-home\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:40:52.441907 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:40:52.441826    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81d123ff-92f3-40a2-a782-2240a40bf0a2-model-cache\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:40:52.443966 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:40:52.443932    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81d123ff-92f3-40a2-a782-2240a40bf0a2-kube-api-access-5j2zh" (OuterVolumeSpecName: "kube-api-access-5j2zh") pod "81d123ff-92f3-40a2-a782-2240a40bf0a2" (UID: "81d123ff-92f3-40a2-a782-2240a40bf0a2"). InnerVolumeSpecName "kube-api-access-5j2zh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:40:52.444069 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:40:52.443964    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/81d123ff-92f3-40a2-a782-2240a40bf0a2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "81d123ff-92f3-40a2-a782-2240a40bf0a2" (UID: "81d123ff-92f3-40a2-a782-2240a40bf0a2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:40:52.444069 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:40:52.443969    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81d123ff-92f3-40a2-a782-2240a40bf0a2-dshm" (OuterVolumeSpecName: "dshm") pod "81d123ff-92f3-40a2-a782-2240a40bf0a2" (UID: "81d123ff-92f3-40a2-a782-2240a40bf0a2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:40:52.457740 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:40:52.457695    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81d123ff-92f3-40a2-a782-2240a40bf0a2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "81d123ff-92f3-40a2-a782-2240a40bf0a2" (UID: "81d123ff-92f3-40a2-a782-2240a40bf0a2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:40:52.464388 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:40:52.464361    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81d123ff-92f3-40a2-a782-2240a40bf0a2-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "81d123ff-92f3-40a2-a782-2240a40bf0a2" (UID: "81d123ff-92f3-40a2-a782-2240a40bf0a2"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:40:52.543281 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:40:52.543245    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81d123ff-92f3-40a2-a782-2240a40bf0a2-dshm\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:40:52.543281 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:40:52.543276    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/81d123ff-92f3-40a2-a782-2240a40bf0a2-cabundle-cert\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:40:52.543281 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:40:52.543287    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5j2zh\" (UniqueName: \"kubernetes.io/projected/81d123ff-92f3-40a2-a782-2240a40bf0a2-kube-api-access-5j2zh\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:40:52.543531 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:40:52.543297    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81d123ff-92f3-40a2-a782-2240a40bf0a2-tls-certs\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:40:52.543531 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:40:52.543305    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81d123ff-92f3-40a2-a782-2240a40bf0a2-tmp-dir\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:40:52.543531 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:40:52.543314    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/81d123ff-92f3-40a2-a782-2240a40bf0a2-kserve-provision-location\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:40:52.579523 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:40:52.579459    2583 generic.go:356] "Generic (PLEG): container finished" podID="81d123ff-92f3-40a2-a782-2240a40bf0a2" containerID="42114d387b7129a6aeda237adc2cb88f201df7bb1375ba98d041f2a9630275f0" exitCode=0
Aug 04 18:40:52.579523 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:40:52.579528    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-66c57bb459-ms2z7"
Aug 04 18:40:52.579767 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:40:52.579532    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-66c57bb459-ms2z7" event={"ID":"81d123ff-92f3-40a2-a782-2240a40bf0a2","Type":"ContainerDied","Data":"42114d387b7129a6aeda237adc2cb88f201df7bb1375ba98d041f2a9630275f0"}
Aug 04 18:40:52.579767 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:40:52.579595    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-66c57bb459-ms2z7" event={"ID":"81d123ff-92f3-40a2-a782-2240a40bf0a2","Type":"ContainerDied","Data":"6d190c7098034950797ce4b8886945d4ac6b7234b82634d7227b877136e0ac5c"}
Aug 04 18:40:52.579767 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:40:52.579624    2583 scope.go:117] "RemoveContainer" containerID="42114d387b7129a6aeda237adc2cb88f201df7bb1375ba98d041f2a9630275f0"
Aug 04 18:40:52.589640 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:40:52.589620    2583 scope.go:117] "RemoveContainer" containerID="7040d4179cc66ff86e8ee40bc562985f34c0dcf9aa0a6e2500cbf1dee9928a15"
Aug 04 18:40:52.599059 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:40:52.599040    2583 scope.go:117] "RemoveContainer" containerID="42114d387b7129a6aeda237adc2cb88f201df7bb1375ba98d041f2a9630275f0"
Aug 04 18:40:52.599296 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:40:52.599276    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42114d387b7129a6aeda237adc2cb88f201df7bb1375ba98d041f2a9630275f0\": container with ID starting with 42114d387b7129a6aeda237adc2cb88f201df7bb1375ba98d041f2a9630275f0 not found: ID does not exist" containerID="42114d387b7129a6aeda237adc2cb88f201df7bb1375ba98d041f2a9630275f0"
Aug 04 18:40:52.599365 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:40:52.599304    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42114d387b7129a6aeda237adc2cb88f201df7bb1375ba98d041f2a9630275f0"} err="failed to get container status \"42114d387b7129a6aeda237adc2cb88f201df7bb1375ba98d041f2a9630275f0\": rpc error: code = NotFound desc = could not find container \"42114d387b7129a6aeda237adc2cb88f201df7bb1375ba98d041f2a9630275f0\": container with ID starting with 42114d387b7129a6aeda237adc2cb88f201df7bb1375ba98d041f2a9630275f0 not found: ID does not exist"
Aug 04 18:40:52.599365 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:40:52.599323    2583 scope.go:117] "RemoveContainer" containerID="7040d4179cc66ff86e8ee40bc562985f34c0dcf9aa0a6e2500cbf1dee9928a15"
Aug 04 18:40:52.599597 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:40:52.599578    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7040d4179cc66ff86e8ee40bc562985f34c0dcf9aa0a6e2500cbf1dee9928a15\": container with ID starting with 7040d4179cc66ff86e8ee40bc562985f34c0dcf9aa0a6e2500cbf1dee9928a15 not found: ID does not exist" containerID="7040d4179cc66ff86e8ee40bc562985f34c0dcf9aa0a6e2500cbf1dee9928a15"
Aug 04 18:40:52.599656 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:40:52.599604    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7040d4179cc66ff86e8ee40bc562985f34c0dcf9aa0a6e2500cbf1dee9928a15"} err="failed to get container status \"7040d4179cc66ff86e8ee40bc562985f34c0dcf9aa0a6e2500cbf1dee9928a15\": rpc error: code = NotFound desc = could not find container \"7040d4179cc66ff86e8ee40bc562985f34c0dcf9aa0a6e2500cbf1dee9928a15\": container with ID starting with 7040d4179cc66ff86e8ee40bc562985f34c0dcf9aa0a6e2500cbf1dee9928a15 not found: ID does not exist"
Aug 04 18:40:52.603459 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:40:52.603438    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-66c57bb459-ms2z7"]
Aug 04 18:40:52.611379 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:40:52.611354    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-66c57bb459-ms2z7"]
Aug 04 18:40:52.942202 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:40:52.942168    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="81d123ff-92f3-40a2-a782-2240a40bf0a2" path="/var/lib/kubelet/pods/81d123ff-92f3-40a2-a782-2240a40bf0a2/volumes"
Aug 04 18:41:01.392238 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:41:01.392200    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-667bfb4c64-qp9jl"]
Aug 04 18:41:01.392981 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:41:01.392667    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99c1175e-4f66-47fc-acf4-3939ed4223f0" containerName="main"
Aug 04 18:41:01.392981 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:41:01.392687    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="99c1175e-4f66-47fc-acf4-3939ed4223f0" containerName="main"
Aug 04 18:41:01.392981 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:41:01.392722    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81d123ff-92f3-40a2-a782-2240a40bf0a2" containerName="main"
Aug 04 18:41:01.392981 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:41:01.392731    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="81d123ff-92f3-40a2-a782-2240a40bf0a2" containerName="main"
Aug 04 18:41:01.392981 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:41:01.392745    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81d123ff-92f3-40a2-a782-2240a40bf0a2" containerName="storage-initializer"
Aug 04 18:41:01.392981 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:41:01.392754    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="81d123ff-92f3-40a2-a782-2240a40bf0a2" containerName="storage-initializer"
Aug 04 18:41:01.392981 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:41:01.392878    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="99c1175e-4f66-47fc-acf4-3939ed4223f0" containerName="main"
Aug 04 18:41:01.392981 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:41:01.392891    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="81d123ff-92f3-40a2-a782-2240a40bf0a2" containerName="main"
Aug 04 18:41:01.396234 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:41:01.396218    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-667bfb4c64-qp9jl"
Aug 04 18:41:01.399045 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:41:01.399021    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"seaweedfs-s3-creds\""
Aug 04 18:41:01.399189 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:41:01.399170    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"default-dockercfg-m8672\""
Aug 04 18:41:01.399250 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:41:01.399177    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"llmisv77ff2528d3e9b4972cd9335229fce9f0-kserve-self-signed-certs\""
Aug 04 18:41:01.399250 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:41:01.399235    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"openshift-service-ca.crt\""
Aug 04 18:41:01.399334 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:41:01.399239    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"kube-root-ca.crt\""
Aug 04 18:41:01.399540 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:41:01.399526    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"odh-kserve-custom-ca-bundle\""
Aug 04 18:41:01.408578 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:41:01.408550    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-667bfb4c64-qp9jl"]
Aug 04 18:41:01.525185 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:41:01.525154    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c0b13630-57de-4419-b702-d0f3f20b4148-home\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-667bfb4c64-qp9jl\" (UID: \"c0b13630-57de-4419-b702-d0f3f20b4148\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-667bfb4c64-qp9jl"
Aug 04 18:41:01.525341 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:41:01.525190    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c0b13630-57de-4419-b702-d0f3f20b4148-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-667bfb4c64-qp9jl\" (UID: \"c0b13630-57de-4419-b702-d0f3f20b4148\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-667bfb4c64-qp9jl"
Aug 04 18:41:01.525341 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:41:01.525227    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c0b13630-57de-4419-b702-d0f3f20b4148-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-667bfb4c64-qp9jl\" (UID: \"c0b13630-57de-4419-b702-d0f3f20b4148\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-667bfb4c64-qp9jl"
Aug 04 18:41:01.525341 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:41:01.525278    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c0b13630-57de-4419-b702-d0f3f20b4148-dshm\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-667bfb4c64-qp9jl\" (UID: \"c0b13630-57de-4419-b702-d0f3f20b4148\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-667bfb4c64-qp9jl"
Aug 04 18:41:01.525341 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:41:01.525326    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c0b13630-57de-4419-b702-d0f3f20b4148-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-667bfb4c64-qp9jl\" (UID: \"c0b13630-57de-4419-b702-d0f3f20b4148\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-667bfb4c64-qp9jl"
Aug 04 18:41:01.525501 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:41:01.525344    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmjkp\" (UniqueName: \"kubernetes.io/projected/c0b13630-57de-4419-b702-d0f3f20b4148-kube-api-access-mmjkp\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-667bfb4c64-qp9jl\" (UID: \"c0b13630-57de-4419-b702-d0f3f20b4148\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-667bfb4c64-qp9jl"
Aug 04 18:41:01.525501 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:41:01.525360    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0b13630-57de-4419-b702-d0f3f20b4148-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-667bfb4c64-qp9jl\" (UID: \"c0b13630-57de-4419-b702-d0f3f20b4148\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-667bfb4c64-qp9jl"
Aug 04 18:41:01.525501 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:41:01.525392    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c0b13630-57de-4419-b702-d0f3f20b4148-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-667bfb4c64-qp9jl\" (UID: \"c0b13630-57de-4419-b702-d0f3f20b4148\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-667bfb4c64-qp9jl"
Aug 04 18:41:01.625823 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:41:01.625773    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c0b13630-57de-4419-b702-d0f3f20b4148-home\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-667bfb4c64-qp9jl\" (UID: \"c0b13630-57de-4419-b702-d0f3f20b4148\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-667bfb4c64-qp9jl"
Aug 04 18:41:01.625823 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:41:01.625828    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c0b13630-57de-4419-b702-d0f3f20b4148-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-667bfb4c64-qp9jl\" (UID: \"c0b13630-57de-4419-b702-d0f3f20b4148\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-667bfb4c64-qp9jl"
Aug 04 18:41:01.626056 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:41:01.625869    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c0b13630-57de-4419-b702-d0f3f20b4148-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-667bfb4c64-qp9jl\" (UID: \"c0b13630-57de-4419-b702-d0f3f20b4148\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-667bfb4c64-qp9jl"
Aug 04 18:41:01.626056 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:41:01.625896    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c0b13630-57de-4419-b702-d0f3f20b4148-dshm\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-667bfb4c64-qp9jl\" (UID: \"c0b13630-57de-4419-b702-d0f3f20b4148\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-667bfb4c64-qp9jl"
Aug 04 18:41:01.626056 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:41:01.626009    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c0b13630-57de-4419-b702-d0f3f20b4148-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-667bfb4c64-qp9jl\" (UID: \"c0b13630-57de-4419-b702-d0f3f20b4148\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-667bfb4c64-qp9jl"
Aug 04 18:41:01.626223 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:41:01.626096    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmjkp\" (UniqueName: \"kubernetes.io/projected/c0b13630-57de-4419-b702-d0f3f20b4148-kube-api-access-mmjkp\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-667bfb4c64-qp9jl\" (UID: \"c0b13630-57de-4419-b702-d0f3f20b4148\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-667bfb4c64-qp9jl"
Aug 04 18:41:01.626223 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:41:01.626166    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0b13630-57de-4419-b702-d0f3f20b4148-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-667bfb4c64-qp9jl\" (UID: \"c0b13630-57de-4419-b702-d0f3f20b4148\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-667bfb4c64-qp9jl"
Aug 04 18:41:01.626364 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:41:01.626247    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c0b13630-57de-4419-b702-d0f3f20b4148-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-667bfb4c64-qp9jl\" (UID: \"c0b13630-57de-4419-b702-d0f3f20b4148\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-667bfb4c64-qp9jl"
Aug 04 18:41:01.626768 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:41:01.626409    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c0b13630-57de-4419-b702-d0f3f20b4148-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-667bfb4c64-qp9jl\" (UID: \"c0b13630-57de-4419-b702-d0f3f20b4148\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-667bfb4c64-qp9jl"
Aug 04 18:41:01.626768 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:41:01.626563    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c0b13630-57de-4419-b702-d0f3f20b4148-home\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-667bfb4c64-qp9jl\" (UID: \"c0b13630-57de-4419-b702-d0f3f20b4148\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-667bfb4c64-qp9jl"
Aug 04 18:41:01.626768 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:41:01.626605    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c0b13630-57de-4419-b702-d0f3f20b4148-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-667bfb4c64-qp9jl\" (UID: \"c0b13630-57de-4419-b702-d0f3f20b4148\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-667bfb4c64-qp9jl"
Aug 04 18:41:01.626768 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:41:01.626726    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0b13630-57de-4419-b702-d0f3f20b4148-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-667bfb4c64-qp9jl\" (UID: \"c0b13630-57de-4419-b702-d0f3f20b4148\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-667bfb4c64-qp9jl"
Aug 04 18:41:01.627058 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:41:01.626877    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c0b13630-57de-4419-b702-d0f3f20b4148-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-667bfb4c64-qp9jl\" (UID: \"c0b13630-57de-4419-b702-d0f3f20b4148\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-667bfb4c64-qp9jl"
Aug 04 18:41:01.628316 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:41:01.628287    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c0b13630-57de-4419-b702-d0f3f20b4148-dshm\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-667bfb4c64-qp9jl\" (UID: \"c0b13630-57de-4419-b702-d0f3f20b4148\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-667bfb4c64-qp9jl"
Aug 04 18:41:01.629217 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:41:01.629192    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c0b13630-57de-4419-b702-d0f3f20b4148-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-667bfb4c64-qp9jl\" (UID: \"c0b13630-57de-4419-b702-d0f3f20b4148\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-667bfb4c64-qp9jl"
Aug 04 18:41:01.635030 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:41:01.635010    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmjkp\" (UniqueName: \"kubernetes.io/projected/c0b13630-57de-4419-b702-d0f3f20b4148-kube-api-access-mmjkp\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-667bfb4c64-qp9jl\" (UID: \"c0b13630-57de-4419-b702-d0f3f20b4148\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-667bfb4c64-qp9jl"
Aug 04 18:41:01.707790 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:41:01.707767    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-667bfb4c64-qp9jl"
Aug 04 18:41:01.846828 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:41:01.846783    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-667bfb4c64-qp9jl"]
Aug 04 18:41:01.848799 ip-10-0-135-124 kubenswrapper[2583]: W0804 18:41:01.848768    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0b13630_57de_4419_b702_d0f3f20b4148.slice/crio-a019d8f02cbc5e3f2bfc76e7931c9a917fadb65ba7c58dd0708cfb39a381abbf WatchSource:0}: Error finding container a019d8f02cbc5e3f2bfc76e7931c9a917fadb65ba7c58dd0708cfb39a381abbf: Status 404 returned error can't find the container with id a019d8f02cbc5e3f2bfc76e7931c9a917fadb65ba7c58dd0708cfb39a381abbf
Aug 04 18:41:02.637922 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:41:02.637878    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-667bfb4c64-qp9jl" event={"ID":"c0b13630-57de-4419-b702-d0f3f20b4148","Type":"ContainerStarted","Data":"5748281f40b4380ec5d0ee45c61aab78a1a30bbd68db31161b60c0c736f84bd7"}
Aug 04 18:41:02.638377 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:41:02.637950    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-667bfb4c64-qp9jl" event={"ID":"c0b13630-57de-4419-b702-d0f3f20b4148","Type":"ContainerStarted","Data":"a019d8f02cbc5e3f2bfc76e7931c9a917fadb65ba7c58dd0708cfb39a381abbf"}
Aug 04 18:41:04.652109 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:41:04.652082    2583 generic.go:356] "Generic (PLEG): container finished" podID="c0b13630-57de-4419-b702-d0f3f20b4148" containerID="5748281f40b4380ec5d0ee45c61aab78a1a30bbd68db31161b60c0c736f84bd7" exitCode=0
Aug 04 18:41:04.652435 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:41:04.652134    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-667bfb4c64-qp9jl" event={"ID":"c0b13630-57de-4419-b702-d0f3f20b4148","Type":"ContainerDied","Data":"5748281f40b4380ec5d0ee45c61aab78a1a30bbd68db31161b60c0c736f84bd7"}
Aug 04 18:41:05.660610 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:41:05.660567    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-667bfb4c64-qp9jl" event={"ID":"c0b13630-57de-4419-b702-d0f3f20b4148","Type":"ContainerStarted","Data":"7a45e1b2b09755584d19472ab8f694ba3ff80f73da128bd0ce28cc0cc792ee54"}
Aug 04 18:41:05.688808 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:41:05.688744    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-667bfb4c64-qp9jl" podStartSLOduration=1.979186301 podStartE2EDuration="4.688726115s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.709539814s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:41:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:41:05.686981479 +0000 UTC m=+2479.268200057" watchObservedRunningTime="2026-08-04 18:41:05.688726115 +0000 UTC m=+2479.269944695"
Aug 04 18:41:05.708502 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:41:05.708460    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-667bfb4c64-qp9jl"
Aug 04 18:41:11.708712 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:41:11.708679    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-667bfb4c64-qp9jl"
Aug 04 18:41:11.710587 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:41:11.710558    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-667bfb4c64-qp9jl" podUID="c0b13630-57de-4419-b702-d0f3f20b4148" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/health\": dial tcp 10.132.0.78:8000: connect: connection refused"
Aug 04 18:41:21.708668 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:41:21.708614    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-667bfb4c64-qp9jl" podUID="c0b13630-57de-4419-b702-d0f3f20b4148" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/health\": dial tcp 10.132.0.78:8000: connect: connection refused"
Aug 04 18:41:31.708973 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:41:31.708929    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-667bfb4c64-qp9jl" podUID="c0b13630-57de-4419-b702-d0f3f20b4148" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/health\": dial tcp 10.132.0.78:8000: connect: connection refused"
Aug 04 18:41:41.708901 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:41:41.708858    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-667bfb4c64-qp9jl" podUID="c0b13630-57de-4419-b702-d0f3f20b4148" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/health\": dial tcp 10.132.0.78:8000: connect: connection refused"
Aug 04 18:41:51.708995 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:41:51.708936    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-667bfb4c64-qp9jl" podUID="c0b13630-57de-4419-b702-d0f3f20b4148" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/health\": dial tcp 10.132.0.78:8000: connect: connection refused"
Aug 04 18:42:01.709094 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:42:01.709046    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-667bfb4c64-qp9jl" podUID="c0b13630-57de-4419-b702-d0f3f20b4148" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/health\": dial tcp 10.132.0.78:8000: connect: connection refused"
Aug 04 18:42:11.708439 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:42:11.708392    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-667bfb4c64-qp9jl" podUID="c0b13630-57de-4419-b702-d0f3f20b4148" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/health\": dial tcp 10.132.0.78:8000: connect: connection refused"
Aug 04 18:42:21.718866 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:42:21.718832    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-667bfb4c64-qp9jl"
Aug 04 18:42:21.727131 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:42:21.727104    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-667bfb4c64-qp9jl"
Aug 04 18:42:26.174496 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:42:26.174391    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-667bfb4c64-qp9jl"]
Aug 04 18:42:41.197555 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:42:41.197512    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-667bfb4c64-qp9jl" podUID="c0b13630-57de-4419-b702-d0f3f20b4148" containerName="main" containerID="cri-o://7a45e1b2b09755584d19472ab8f694ba3ff80f73da128bd0ce28cc0cc792ee54" gracePeriod=45
Aug 04 18:42:41.752368 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:42:41.752340    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-667bfb4c64-qp9jl"
Aug 04 18:42:41.903337 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:42:41.903255    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c0b13630-57de-4419-b702-d0f3f20b4148-tls-certs\") pod \"c0b13630-57de-4419-b702-d0f3f20b4148\" (UID: \"c0b13630-57de-4419-b702-d0f3f20b4148\") "
Aug 04 18:42:41.903337 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:42:41.903307    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c0b13630-57de-4419-b702-d0f3f20b4148-dshm\") pod \"c0b13630-57de-4419-b702-d0f3f20b4148\" (UID: \"c0b13630-57de-4419-b702-d0f3f20b4148\") "
Aug 04 18:42:41.903337 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:42:41.903338    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c0b13630-57de-4419-b702-d0f3f20b4148-home\") pod \"c0b13630-57de-4419-b702-d0f3f20b4148\" (UID: \"c0b13630-57de-4419-b702-d0f3f20b4148\") "
Aug 04 18:42:41.903662 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:42:41.903365    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mmjkp\" (UniqueName: \"kubernetes.io/projected/c0b13630-57de-4419-b702-d0f3f20b4148-kube-api-access-mmjkp\") pod \"c0b13630-57de-4419-b702-d0f3f20b4148\" (UID: \"c0b13630-57de-4419-b702-d0f3f20b4148\") "
Aug 04 18:42:41.903662 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:42:41.903392    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c0b13630-57de-4419-b702-d0f3f20b4148-model-cache\") pod \"c0b13630-57de-4419-b702-d0f3f20b4148\" (UID: \"c0b13630-57de-4419-b702-d0f3f20b4148\") "
Aug 04 18:42:41.903662 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:42:41.903444    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c0b13630-57de-4419-b702-d0f3f20b4148-cabundle-cert\") pod \"c0b13630-57de-4419-b702-d0f3f20b4148\" (UID: \"c0b13630-57de-4419-b702-d0f3f20b4148\") "
Aug 04 18:42:41.903662 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:42:41.903640    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0b13630-57de-4419-b702-d0f3f20b4148-tmp-dir\") pod \"c0b13630-57de-4419-b702-d0f3f20b4148\" (UID: \"c0b13630-57de-4419-b702-d0f3f20b4148\") "
Aug 04 18:42:41.903901 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:42:41.903708    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0b13630-57de-4419-b702-d0f3f20b4148-model-cache" (OuterVolumeSpecName: "model-cache") pod "c0b13630-57de-4419-b702-d0f3f20b4148" (UID: "c0b13630-57de-4419-b702-d0f3f20b4148"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:42:41.903901 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:42:41.903737    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c0b13630-57de-4419-b702-d0f3f20b4148-kserve-provision-location\") pod \"c0b13630-57de-4419-b702-d0f3f20b4148\" (UID: \"c0b13630-57de-4419-b702-d0f3f20b4148\") "
Aug 04 18:42:41.903901 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:42:41.903837    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0b13630-57de-4419-b702-d0f3f20b4148-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "c0b13630-57de-4419-b702-d0f3f20b4148" (UID: "c0b13630-57de-4419-b702-d0f3f20b4148"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:42:41.904143 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:42:41.903841    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0b13630-57de-4419-b702-d0f3f20b4148-home" (OuterVolumeSpecName: "home") pod "c0b13630-57de-4419-b702-d0f3f20b4148" (UID: "c0b13630-57de-4419-b702-d0f3f20b4148"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:42:41.904143 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:42:41.904124    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c0b13630-57de-4419-b702-d0f3f20b4148-model-cache\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:42:41.904143 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:42:41.904128    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0b13630-57de-4419-b702-d0f3f20b4148-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c0b13630-57de-4419-b702-d0f3f20b4148" (UID: "c0b13630-57de-4419-b702-d0f3f20b4148"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:42:41.904328 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:42:41.904145    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c0b13630-57de-4419-b702-d0f3f20b4148-cabundle-cert\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:42:41.904328 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:42:41.904158    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c0b13630-57de-4419-b702-d0f3f20b4148-home\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:42:41.905758 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:42:41.905732    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0b13630-57de-4419-b702-d0f3f20b4148-dshm" (OuterVolumeSpecName: "dshm") pod "c0b13630-57de-4419-b702-d0f3f20b4148" (UID: "c0b13630-57de-4419-b702-d0f3f20b4148"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:42:41.905854 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:42:41.905822    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0b13630-57de-4419-b702-d0f3f20b4148-kube-api-access-mmjkp" (OuterVolumeSpecName: "kube-api-access-mmjkp") pod "c0b13630-57de-4419-b702-d0f3f20b4148" (UID: "c0b13630-57de-4419-b702-d0f3f20b4148"). InnerVolumeSpecName "kube-api-access-mmjkp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:42:41.905931 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:42:41.905895    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0b13630-57de-4419-b702-d0f3f20b4148-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c0b13630-57de-4419-b702-d0f3f20b4148" (UID: "c0b13630-57de-4419-b702-d0f3f20b4148"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:42:41.927745 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:42:41.927716    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0b13630-57de-4419-b702-d0f3f20b4148-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c0b13630-57de-4419-b702-d0f3f20b4148" (UID: "c0b13630-57de-4419-b702-d0f3f20b4148"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:42:42.004887 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:42:42.004840    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0b13630-57de-4419-b702-d0f3f20b4148-tmp-dir\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:42:42.004887 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:42:42.004883    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c0b13630-57de-4419-b702-d0f3f20b4148-kserve-provision-location\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:42:42.004887 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:42:42.004897    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c0b13630-57de-4419-b702-d0f3f20b4148-tls-certs\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:42:42.005091 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:42:42.004910    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c0b13630-57de-4419-b702-d0f3f20b4148-dshm\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:42:42.005091 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:42:42.004921    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mmjkp\" (UniqueName: \"kubernetes.io/projected/c0b13630-57de-4419-b702-d0f3f20b4148-kube-api-access-mmjkp\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:42:42.187992 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:42:42.187956    2583 generic.go:356] "Generic (PLEG): container finished" podID="c0b13630-57de-4419-b702-d0f3f20b4148" containerID="7a45e1b2b09755584d19472ab8f694ba3ff80f73da128bd0ce28cc0cc792ee54" exitCode=0
Aug 04 18:42:42.188156 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:42:42.187996    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-667bfb4c64-qp9jl" event={"ID":"c0b13630-57de-4419-b702-d0f3f20b4148","Type":"ContainerDied","Data":"7a45e1b2b09755584d19472ab8f694ba3ff80f73da128bd0ce28cc0cc792ee54"}
Aug 04 18:42:42.188156 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:42:42.188019    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-667bfb4c64-qp9jl"
Aug 04 18:42:42.188156 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:42:42.188066    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-667bfb4c64-qp9jl" event={"ID":"c0b13630-57de-4419-b702-d0f3f20b4148","Type":"ContainerDied","Data":"a019d8f02cbc5e3f2bfc76e7931c9a917fadb65ba7c58dd0708cfb39a381abbf"}
Aug 04 18:42:42.188156 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:42:42.188097    2583 scope.go:117] "RemoveContainer" containerID="7a45e1b2b09755584d19472ab8f694ba3ff80f73da128bd0ce28cc0cc792ee54"
Aug 04 18:42:42.197627 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:42:42.197610    2583 scope.go:117] "RemoveContainer" containerID="5748281f40b4380ec5d0ee45c61aab78a1a30bbd68db31161b60c0c736f84bd7"
Aug 04 18:42:42.206218 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:42:42.206199    2583 scope.go:117] "RemoveContainer" containerID="7a45e1b2b09755584d19472ab8f694ba3ff80f73da128bd0ce28cc0cc792ee54"
Aug 04 18:42:42.206509 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:42:42.206464    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a45e1b2b09755584d19472ab8f694ba3ff80f73da128bd0ce28cc0cc792ee54\": container with ID starting with 7a45e1b2b09755584d19472ab8f694ba3ff80f73da128bd0ce28cc0cc792ee54 not found: ID does not exist" containerID="7a45e1b2b09755584d19472ab8f694ba3ff80f73da128bd0ce28cc0cc792ee54"
Aug 04 18:42:42.206590 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:42:42.206517    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a45e1b2b09755584d19472ab8f694ba3ff80f73da128bd0ce28cc0cc792ee54"} err="failed to get container status \"7a45e1b2b09755584d19472ab8f694ba3ff80f73da128bd0ce28cc0cc792ee54\": rpc error: code = NotFound desc = could not find container \"7a45e1b2b09755584d19472ab8f694ba3ff80f73da128bd0ce28cc0cc792ee54\": container with ID starting with 7a45e1b2b09755584d19472ab8f694ba3ff80f73da128bd0ce28cc0cc792ee54 not found: ID does not exist"
Aug 04 18:42:42.206590 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:42:42.206534    2583 scope.go:117] "RemoveContainer" containerID="5748281f40b4380ec5d0ee45c61aab78a1a30bbd68db31161b60c0c736f84bd7"
Aug 04 18:42:42.206773 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:42:42.206754    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5748281f40b4380ec5d0ee45c61aab78a1a30bbd68db31161b60c0c736f84bd7\": container with ID starting with 5748281f40b4380ec5d0ee45c61aab78a1a30bbd68db31161b60c0c736f84bd7 not found: ID does not exist" containerID="5748281f40b4380ec5d0ee45c61aab78a1a30bbd68db31161b60c0c736f84bd7"
Aug 04 18:42:42.206833 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:42:42.206782    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5748281f40b4380ec5d0ee45c61aab78a1a30bbd68db31161b60c0c736f84bd7"} err="failed to get container status \"5748281f40b4380ec5d0ee45c61aab78a1a30bbd68db31161b60c0c736f84bd7\": rpc error: code = NotFound desc = could not find container \"5748281f40b4380ec5d0ee45c61aab78a1a30bbd68db31161b60c0c736f84bd7\": container with ID starting with 5748281f40b4380ec5d0ee45c61aab78a1a30bbd68db31161b60c0c736f84bd7 not found: ID does not exist"
Aug 04 18:42:42.214620 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:42:42.214598    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-667bfb4c64-qp9jl"]
Aug 04 18:42:42.220839 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:42:42.220819    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-667bfb4c64-qp9jl"]
Aug 04 18:42:42.720583 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:42:42.720543    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-667bfb4c64-qp9jl" podUID="c0b13630-57de-4419-b702-d0f3f20b4148" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Aug 04 18:42:42.942128 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:42:42.942095    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c0b13630-57de-4419-b702-d0f3f20b4148" path="/var/lib/kubelet/pods/c0b13630-57de-4419-b702-d0f3f20b4148/volumes"
Aug 04 18:43:12.110796 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:43:12.110717    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6b5bb78f5-s2ftw"]
Aug 04 18:43:12.111245 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:43:12.111103    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0b13630-57de-4419-b702-d0f3f20b4148" containerName="storage-initializer"
Aug 04 18:43:12.111245 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:43:12.111117    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0b13630-57de-4419-b702-d0f3f20b4148" containerName="storage-initializer"
Aug 04 18:43:12.111245 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:43:12.111125    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0b13630-57de-4419-b702-d0f3f20b4148" containerName="main"
Aug 04 18:43:12.111245 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:43:12.111130    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0b13630-57de-4419-b702-d0f3f20b4148" containerName="main"
Aug 04 18:43:12.111245 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:43:12.111187    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="c0b13630-57de-4419-b702-d0f3f20b4148" containerName="main"
Aug 04 18:43:12.114089 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:43:12.114071    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6b5bb78f5-s2ftw"
Aug 04 18:43:12.117013 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:43:12.116984    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-config-deletion-blocked-when-referenced-0f029b85\"/\"default-dockercfg-5b8h5\""
Aug 04 18:43:12.117013 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:43:12.116984    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-config-deletion-blocked-when-referenced-0f029b85\"/\"kube-root-ca.crt\""
Aug 04 18:43:12.117192 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:43:12.117007    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-config-deletion-blocked-when-referenced-0f029b85\"/\"seaweedfs-s3-creds\""
Aug 04 18:43:12.117192 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:43:12.117176    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-config-deletion-blocked-when-referenced-0f029b85\"/\"e2e-del-blocked-svc-kserve-self-signed-certs\""
Aug 04 18:43:12.117292 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:43:12.117258    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-config-deletion-blocked-when-referenced-0f029b85\"/\"openshift-service-ca.crt\""
Aug 04 18:43:12.128433 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:43:12.128373    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6b5bb78f5-s2ftw"]
Aug 04 18:43:12.158285 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:43:12.158257    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90f00c4c-4a2a-472c-bcac-e5518414b13b-home\") pod \"e2e-del-blocked-svc-kserve-6b5bb78f5-s2ftw\" (UID: \"90f00c4c-4a2a-472c-bcac-e5518414b13b\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6b5bb78f5-s2ftw"
Aug 04 18:43:12.158404 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:43:12.158290    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/90f00c4c-4a2a-472c-bcac-e5518414b13b-kserve-provision-location\") pod \"e2e-del-blocked-svc-kserve-6b5bb78f5-s2ftw\" (UID: \"90f00c4c-4a2a-472c-bcac-e5518414b13b\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6b5bb78f5-s2ftw"
Aug 04 18:43:12.158404 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:43:12.158314    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90f00c4c-4a2a-472c-bcac-e5518414b13b-tmp-dir\") pod \"e2e-del-blocked-svc-kserve-6b5bb78f5-s2ftw\" (UID: \"90f00c4c-4a2a-472c-bcac-e5518414b13b\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6b5bb78f5-s2ftw"
Aug 04 18:43:12.158499 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:43:12.158399    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90f00c4c-4a2a-472c-bcac-e5518414b13b-tls-certs\") pod \"e2e-del-blocked-svc-kserve-6b5bb78f5-s2ftw\" (UID: \"90f00c4c-4a2a-472c-bcac-e5518414b13b\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6b5bb78f5-s2ftw"
Aug 04 18:43:12.158499 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:43:12.158433    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55rnf\" (UniqueName: \"kubernetes.io/projected/90f00c4c-4a2a-472c-bcac-e5518414b13b-kube-api-access-55rnf\") pod \"e2e-del-blocked-svc-kserve-6b5bb78f5-s2ftw\" (UID: \"90f00c4c-4a2a-472c-bcac-e5518414b13b\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6b5bb78f5-s2ftw"
Aug 04 18:43:12.158499 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:43:12.158458    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90f00c4c-4a2a-472c-bcac-e5518414b13b-model-cache\") pod \"e2e-del-blocked-svc-kserve-6b5bb78f5-s2ftw\" (UID: \"90f00c4c-4a2a-472c-bcac-e5518414b13b\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6b5bb78f5-s2ftw"
Aug 04 18:43:12.158625 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:43:12.158517    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90f00c4c-4a2a-472c-bcac-e5518414b13b-dshm\") pod \"e2e-del-blocked-svc-kserve-6b5bb78f5-s2ftw\" (UID: \"90f00c4c-4a2a-472c-bcac-e5518414b13b\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6b5bb78f5-s2ftw"
Aug 04 18:43:12.259411 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:43:12.259380    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90f00c4c-4a2a-472c-bcac-e5518414b13b-model-cache\") pod \"e2e-del-blocked-svc-kserve-6b5bb78f5-s2ftw\" (UID: \"90f00c4c-4a2a-472c-bcac-e5518414b13b\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6b5bb78f5-s2ftw"
Aug 04 18:43:12.259577 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:43:12.259426    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90f00c4c-4a2a-472c-bcac-e5518414b13b-dshm\") pod \"e2e-del-blocked-svc-kserve-6b5bb78f5-s2ftw\" (UID: \"90f00c4c-4a2a-472c-bcac-e5518414b13b\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6b5bb78f5-s2ftw"
Aug 04 18:43:12.259577 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:43:12.259487    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90f00c4c-4a2a-472c-bcac-e5518414b13b-home\") pod \"e2e-del-blocked-svc-kserve-6b5bb78f5-s2ftw\" (UID: \"90f00c4c-4a2a-472c-bcac-e5518414b13b\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6b5bb78f5-s2ftw"
Aug 04 18:43:12.259577 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:43:12.259518    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/90f00c4c-4a2a-472c-bcac-e5518414b13b-kserve-provision-location\") pod \"e2e-del-blocked-svc-kserve-6b5bb78f5-s2ftw\" (UID: \"90f00c4c-4a2a-472c-bcac-e5518414b13b\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6b5bb78f5-s2ftw"
Aug 04 18:43:12.259577 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:43:12.259549    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90f00c4c-4a2a-472c-bcac-e5518414b13b-tmp-dir\") pod \"e2e-del-blocked-svc-kserve-6b5bb78f5-s2ftw\" (UID: \"90f00c4c-4a2a-472c-bcac-e5518414b13b\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6b5bb78f5-s2ftw"
Aug 04 18:43:12.259785 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:43:12.259591    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90f00c4c-4a2a-472c-bcac-e5518414b13b-tls-certs\") pod \"e2e-del-blocked-svc-kserve-6b5bb78f5-s2ftw\" (UID: \"90f00c4c-4a2a-472c-bcac-e5518414b13b\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6b5bb78f5-s2ftw"
Aug 04 18:43:12.259785 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:43:12.259618    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55rnf\" (UniqueName: \"kubernetes.io/projected/90f00c4c-4a2a-472c-bcac-e5518414b13b-kube-api-access-55rnf\") pod \"e2e-del-blocked-svc-kserve-6b5bb78f5-s2ftw\" (UID: \"90f00c4c-4a2a-472c-bcac-e5518414b13b\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6b5bb78f5-s2ftw"
Aug 04 18:43:12.259903 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:43:12.259806    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90f00c4c-4a2a-472c-bcac-e5518414b13b-model-cache\") pod \"e2e-del-blocked-svc-kserve-6b5bb78f5-s2ftw\" (UID: \"90f00c4c-4a2a-472c-bcac-e5518414b13b\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6b5bb78f5-s2ftw"
Aug 04 18:43:12.259994 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:43:12.259972    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/90f00c4c-4a2a-472c-bcac-e5518414b13b-kserve-provision-location\") pod \"e2e-del-blocked-svc-kserve-6b5bb78f5-s2ftw\" (UID: \"90f00c4c-4a2a-472c-bcac-e5518414b13b\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6b5bb78f5-s2ftw"
Aug 04 18:43:12.260090 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:43:12.260069    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90f00c4c-4a2a-472c-bcac-e5518414b13b-tmp-dir\") pod \"e2e-del-blocked-svc-kserve-6b5bb78f5-s2ftw\" (UID: \"90f00c4c-4a2a-472c-bcac-e5518414b13b\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6b5bb78f5-s2ftw"
Aug 04 18:43:12.260132 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:43:12.260103    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90f00c4c-4a2a-472c-bcac-e5518414b13b-home\") pod \"e2e-del-blocked-svc-kserve-6b5bb78f5-s2ftw\" (UID: \"90f00c4c-4a2a-472c-bcac-e5518414b13b\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6b5bb78f5-s2ftw"
Aug 04 18:43:12.261711 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:43:12.261690    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90f00c4c-4a2a-472c-bcac-e5518414b13b-dshm\") pod \"e2e-del-blocked-svc-kserve-6b5bb78f5-s2ftw\" (UID: \"90f00c4c-4a2a-472c-bcac-e5518414b13b\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6b5bb78f5-s2ftw"
Aug 04 18:43:12.262050 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:43:12.262029    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90f00c4c-4a2a-472c-bcac-e5518414b13b-tls-certs\") pod \"e2e-del-blocked-svc-kserve-6b5bb78f5-s2ftw\" (UID: \"90f00c4c-4a2a-472c-bcac-e5518414b13b\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6b5bb78f5-s2ftw"
Aug 04 18:43:12.267877 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:43:12.267847    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-55rnf\" (UniqueName: \"kubernetes.io/projected/90f00c4c-4a2a-472c-bcac-e5518414b13b-kube-api-access-55rnf\") pod \"e2e-del-blocked-svc-kserve-6b5bb78f5-s2ftw\" (UID: \"90f00c4c-4a2a-472c-bcac-e5518414b13b\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6b5bb78f5-s2ftw"
Aug 04 18:43:12.424787 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:43:12.424706    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6b5bb78f5-s2ftw"
Aug 04 18:43:12.566092 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:43:12.566061    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6b5bb78f5-s2ftw"]
Aug 04 18:43:12.568679 ip-10-0-135-124 kubenswrapper[2583]: W0804 18:43:12.568649    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90f00c4c_4a2a_472c_bcac_e5518414b13b.slice/crio-f1b4497ad7e8202c1400589271419c0da935708ac621664f6d630ebc5e29469e WatchSource:0}: Error finding container f1b4497ad7e8202c1400589271419c0da935708ac621664f6d630ebc5e29469e: Status 404 returned error can't find the container with id f1b4497ad7e8202c1400589271419c0da935708ac621664f6d630ebc5e29469e
Aug 04 18:43:13.352373 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:43:13.352328    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6b5bb78f5-s2ftw" event={"ID":"90f00c4c-4a2a-472c-bcac-e5518414b13b","Type":"ContainerStarted","Data":"4009086cfaa3730cd76db418d6b3ed3498363bccb2dfdc13a7fe47d3edebc666"}
Aug 04 18:43:13.352816 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:43:13.352392    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6b5bb78f5-s2ftw" event={"ID":"90f00c4c-4a2a-472c-bcac-e5518414b13b","Type":"ContainerStarted","Data":"f1b4497ad7e8202c1400589271419c0da935708ac621664f6d630ebc5e29469e"}
Aug 04 18:43:17.376377 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:43:17.376294    2583 generic.go:356] "Generic (PLEG): container finished" podID="90f00c4c-4a2a-472c-bcac-e5518414b13b" containerID="4009086cfaa3730cd76db418d6b3ed3498363bccb2dfdc13a7fe47d3edebc666" exitCode=0
Aug 04 18:43:17.376754 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:43:17.376368    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6b5bb78f5-s2ftw" event={"ID":"90f00c4c-4a2a-472c-bcac-e5518414b13b","Type":"ContainerDied","Data":"4009086cfaa3730cd76db418d6b3ed3498363bccb2dfdc13a7fe47d3edebc666"}
Aug 04 18:43:18.383852 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:43:18.383815    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6b5bb78f5-s2ftw" event={"ID":"90f00c4c-4a2a-472c-bcac-e5518414b13b","Type":"ContainerStarted","Data":"8eb467d97c98de9004c87cde569d5aed1c4bd654bdcc15b77759509e4c37c988"}
Aug 04 18:43:18.410371 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:43:18.410321    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6b5bb78f5-s2ftw" podStartSLOduration=2.069602214 podStartE2EDuration="6.410305706s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.340703492s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:43:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:43:18.408723125 +0000 UTC m=+2611.989941717" watchObservedRunningTime="2026-08-04 18:43:18.410305706 +0000 UTC m=+2611.991524285"
Aug 04 18:43:18.425548 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:43:18.425524    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6b5bb78f5-s2ftw"
Aug 04 18:43:22.425096 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:43:22.425050    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6b5bb78f5-s2ftw"
Aug 04 18:43:22.426772 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:43:22.426741    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6b5bb78f5-s2ftw" podUID="90f00c4c-4a2a-472c-bcac-e5518414b13b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/health\": dial tcp 10.132.0.79:8000: connect: connection refused"
Aug 04 18:43:30.247664 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:43:30.247629    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6b5bb78f5-s2ftw"]
Aug 04 18:43:45.274733 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:43:45.274695    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6b5bb78f5-s2ftw" podUID="90f00c4c-4a2a-472c-bcac-e5518414b13b" containerName="main" containerID="cri-o://8eb467d97c98de9004c87cde569d5aed1c4bd654bdcc15b77759509e4c37c988" gracePeriod=45
Aug 04 18:44:30.775414 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:44:30.775384    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-config-deletion-blocked-when-referenced-0f029b85_e2e-del-blocked-svc-kserve-6b5bb78f5-s2ftw_90f00c4c-4a2a-472c-bcac-e5518414b13b/main/0.log"
Aug 04 18:44:30.775857 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:44:30.775772    2583 generic.go:356] "Generic (PLEG): container finished" podID="90f00c4c-4a2a-472c-bcac-e5518414b13b" containerID="8eb467d97c98de9004c87cde569d5aed1c4bd654bdcc15b77759509e4c37c988" exitCode=137
Aug 04 18:44:30.775857 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:44:30.775828    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6b5bb78f5-s2ftw" event={"ID":"90f00c4c-4a2a-472c-bcac-e5518414b13b","Type":"ContainerDied","Data":"8eb467d97c98de9004c87cde569d5aed1c4bd654bdcc15b77759509e4c37c988"}
Aug 04 18:44:30.932573 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:44:30.932549    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-config-deletion-blocked-when-referenced-0f029b85_e2e-del-blocked-svc-kserve-6b5bb78f5-s2ftw_90f00c4c-4a2a-472c-bcac-e5518414b13b/main/0.log"
Aug 04 18:44:30.935194 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:44:30.935174    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6b5bb78f5-s2ftw"
Aug 04 18:44:31.035509 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:44:31.035426    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90f00c4c-4a2a-472c-bcac-e5518414b13b-tmp-dir\") pod \"90f00c4c-4a2a-472c-bcac-e5518414b13b\" (UID: \"90f00c4c-4a2a-472c-bcac-e5518414b13b\") "
Aug 04 18:44:31.035657 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:44:31.035513    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/90f00c4c-4a2a-472c-bcac-e5518414b13b-kserve-provision-location\") pod \"90f00c4c-4a2a-472c-bcac-e5518414b13b\" (UID: \"90f00c4c-4a2a-472c-bcac-e5518414b13b\") "
Aug 04 18:44:31.035657 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:44:31.035543    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90f00c4c-4a2a-472c-bcac-e5518414b13b-model-cache\") pod \"90f00c4c-4a2a-472c-bcac-e5518414b13b\" (UID: \"90f00c4c-4a2a-472c-bcac-e5518414b13b\") "
Aug 04 18:44:31.035657 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:44:31.035561    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90f00c4c-4a2a-472c-bcac-e5518414b13b-dshm\") pod \"90f00c4c-4a2a-472c-bcac-e5518414b13b\" (UID: \"90f00c4c-4a2a-472c-bcac-e5518414b13b\") "
Aug 04 18:44:31.035657 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:44:31.035622    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90f00c4c-4a2a-472c-bcac-e5518414b13b-tls-certs\") pod \"90f00c4c-4a2a-472c-bcac-e5518414b13b\" (UID: \"90f00c4c-4a2a-472c-bcac-e5518414b13b\") "
Aug 04 18:44:31.035868 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:44:31.035660    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90f00c4c-4a2a-472c-bcac-e5518414b13b-home\") pod \"90f00c4c-4a2a-472c-bcac-e5518414b13b\" (UID: \"90f00c4c-4a2a-472c-bcac-e5518414b13b\") "
Aug 04 18:44:31.035868 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:44:31.035685    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-55rnf\" (UniqueName: \"kubernetes.io/projected/90f00c4c-4a2a-472c-bcac-e5518414b13b-kube-api-access-55rnf\") pod \"90f00c4c-4a2a-472c-bcac-e5518414b13b\" (UID: \"90f00c4c-4a2a-472c-bcac-e5518414b13b\") "
Aug 04 18:44:31.035868 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:44:31.035799    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90f00c4c-4a2a-472c-bcac-e5518414b13b-model-cache" (OuterVolumeSpecName: "model-cache") pod "90f00c4c-4a2a-472c-bcac-e5518414b13b" (UID: "90f00c4c-4a2a-472c-bcac-e5518414b13b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:44:31.036019 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:44:31.035976    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90f00c4c-4a2a-472c-bcac-e5518414b13b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "90f00c4c-4a2a-472c-bcac-e5518414b13b" (UID: "90f00c4c-4a2a-472c-bcac-e5518414b13b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:44:31.036076 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:44:31.036043    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90f00c4c-4a2a-472c-bcac-e5518414b13b-tmp-dir\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:44:31.036076 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:44:31.036062    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90f00c4c-4a2a-472c-bcac-e5518414b13b-model-cache\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:44:31.036360 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:44:31.036320    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90f00c4c-4a2a-472c-bcac-e5518414b13b-home" (OuterVolumeSpecName: "home") pod "90f00c4c-4a2a-472c-bcac-e5518414b13b" (UID: "90f00c4c-4a2a-472c-bcac-e5518414b13b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:44:31.038000 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:44:31.037969    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90f00c4c-4a2a-472c-bcac-e5518414b13b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "90f00c4c-4a2a-472c-bcac-e5518414b13b" (UID: "90f00c4c-4a2a-472c-bcac-e5518414b13b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:44:31.038300 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:44:31.038265    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90f00c4c-4a2a-472c-bcac-e5518414b13b-dshm" (OuterVolumeSpecName: "dshm") pod "90f00c4c-4a2a-472c-bcac-e5518414b13b" (UID: "90f00c4c-4a2a-472c-bcac-e5518414b13b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:44:31.038300 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:44:31.038279    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90f00c4c-4a2a-472c-bcac-e5518414b13b-kube-api-access-55rnf" (OuterVolumeSpecName: "kube-api-access-55rnf") pod "90f00c4c-4a2a-472c-bcac-e5518414b13b" (UID: "90f00c4c-4a2a-472c-bcac-e5518414b13b"). InnerVolumeSpecName "kube-api-access-55rnf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:44:31.098860 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:44:31.098810    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90f00c4c-4a2a-472c-bcac-e5518414b13b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "90f00c4c-4a2a-472c-bcac-e5518414b13b" (UID: "90f00c4c-4a2a-472c-bcac-e5518414b13b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:44:31.136990 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:44:31.136965    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/90f00c4c-4a2a-472c-bcac-e5518414b13b-kserve-provision-location\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:44:31.136990 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:44:31.136989    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90f00c4c-4a2a-472c-bcac-e5518414b13b-dshm\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:44:31.137149 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:44:31.137000    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90f00c4c-4a2a-472c-bcac-e5518414b13b-tls-certs\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:44:31.137149 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:44:31.137010    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90f00c4c-4a2a-472c-bcac-e5518414b13b-home\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:44:31.137149 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:44:31.137018    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-55rnf\" (UniqueName: \"kubernetes.io/projected/90f00c4c-4a2a-472c-bcac-e5518414b13b-kube-api-access-55rnf\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:44:31.782192 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:44:31.782161    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-config-deletion-blocked-when-referenced-0f029b85_e2e-del-blocked-svc-kserve-6b5bb78f5-s2ftw_90f00c4c-4a2a-472c-bcac-e5518414b13b/main/0.log"
Aug 04 18:44:31.782709 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:44:31.782555    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6b5bb78f5-s2ftw" event={"ID":"90f00c4c-4a2a-472c-bcac-e5518414b13b","Type":"ContainerDied","Data":"f1b4497ad7e8202c1400589271419c0da935708ac621664f6d630ebc5e29469e"}
Aug 04 18:44:31.782709 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:44:31.782612    2583 scope.go:117] "RemoveContainer" containerID="8eb467d97c98de9004c87cde569d5aed1c4bd654bdcc15b77759509e4c37c988"
Aug 04 18:44:31.782709 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:44:31.782567    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6b5bb78f5-s2ftw"
Aug 04 18:44:31.791770 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:44:31.791749    2583 scope.go:117] "RemoveContainer" containerID="4009086cfaa3730cd76db418d6b3ed3498363bccb2dfdc13a7fe47d3edebc666"
Aug 04 18:44:31.824986 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:44:31.824910    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6b5bb78f5-s2ftw"]
Aug 04 18:44:31.833038 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:44:31.833010    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-6b5bb78f5-s2ftw"]
Aug 04 18:44:32.942747 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:44:32.942714    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="90f00c4c-4a2a-472c-bcac-e5518414b13b" path="/var/lib/kubelet/pods/90f00c4c-4a2a-472c-bcac-e5518414b13b/volumes"
Aug 04 18:44:47.139737 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:44:47.139624    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-wskfb_6157c2d3-2fd0-4544-88ad-434ac75f8c04/console-operator/1.log"
Aug 04 18:44:47.150063 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:44:47.150039    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-wskfb_6157c2d3-2fd0-4544-88ad-434ac75f8c04/console-operator/1.log"
Aug 04 18:45:53.423879 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:45:53.423841    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-579b7ffcc5sdtvq"]
Aug 04 18:45:53.424341 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:45:53.424218    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="90f00c4c-4a2a-472c-bcac-e5518414b13b" containerName="storage-initializer"
Aug 04 18:45:53.424341 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:45:53.424231    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="90f00c4c-4a2a-472c-bcac-e5518414b13b" containerName="storage-initializer"
Aug 04 18:45:53.424341 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:45:53.424252    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="90f00c4c-4a2a-472c-bcac-e5518414b13b" containerName="main"
Aug 04 18:45:53.424341 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:45:53.424258    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="90f00c4c-4a2a-472c-bcac-e5518414b13b" containerName="main"
Aug 04 18:45:53.424341 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:45:53.424334    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="90f00c4c-4a2a-472c-bcac-e5518414b13b" containerName="main"
Aug 04 18:45:53.427461 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:45:53.427431    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-579b7ffcc5sdtvq"
Aug 04 18:45:53.430764 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:45:53.430736    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"openshift-service-ca.crt\""
Aug 04 18:45:53.430914 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:45:53.430878    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"seaweedfs-s3-creds\""
Aug 04 18:45:53.430987 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:45:53.430792    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"odh-kserve-custom-ca-bundle\""
Aug 04 18:45:53.430987 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:45:53.430799    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"kube-root-ca.crt\""
Aug 04 18:45:53.431101 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:45:53.430758    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"llmisva690bbc929faec8bc98c767f16c003c1-kserve-self-signed-certs\""
Aug 04 18:45:53.431160 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:45:53.431146    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"default-dockercfg-mvjvd\""
Aug 04 18:45:53.438560 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:45:53.438535    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-579b7ffcc5sdtvq"]
Aug 04 18:45:53.536373 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:45:53.536327    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c3091da-e8ba-4c2c-9d26-f02d4c5835bb-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-579b7ffcc5sdtvq\" (UID: \"5c3091da-e8ba-4c2c-9d26-f02d4c5835bb\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-579b7ffcc5sdtvq"
Aug 04 18:45:53.536373 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:45:53.536373    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c3091da-e8ba-4c2c-9d26-f02d4c5835bb-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-579b7ffcc5sdtvq\" (UID: \"5c3091da-e8ba-4c2c-9d26-f02d4c5835bb\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-579b7ffcc5sdtvq"
Aug 04 18:45:53.536645 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:45:53.536401    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c3091da-e8ba-4c2c-9d26-f02d4c5835bb-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-579b7ffcc5sdtvq\" (UID: \"5c3091da-e8ba-4c2c-9d26-f02d4c5835bb\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-579b7ffcc5sdtvq"
Aug 04 18:45:53.536645 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:45:53.536529    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5c3091da-e8ba-4c2c-9d26-f02d4c5835bb-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-579b7ffcc5sdtvq\" (UID: \"5c3091da-e8ba-4c2c-9d26-f02d4c5835bb\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-579b7ffcc5sdtvq"
Aug 04 18:45:53.536645 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:45:53.536627    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mmw7\" (UniqueName: \"kubernetes.io/projected/5c3091da-e8ba-4c2c-9d26-f02d4c5835bb-kube-api-access-8mmw7\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-579b7ffcc5sdtvq\" (UID: \"5c3091da-e8ba-4c2c-9d26-f02d4c5835bb\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-579b7ffcc5sdtvq"
Aug 04 18:45:53.536757 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:45:53.536675    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5c3091da-e8ba-4c2c-9d26-f02d4c5835bb-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-579b7ffcc5sdtvq\" (UID: \"5c3091da-e8ba-4c2c-9d26-f02d4c5835bb\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-579b7ffcc5sdtvq"
Aug 04 18:45:53.536757 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:45:53.536697    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c3091da-e8ba-4c2c-9d26-f02d4c5835bb-dshm\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-579b7ffcc5sdtvq\" (UID: \"5c3091da-e8ba-4c2c-9d26-f02d4c5835bb\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-579b7ffcc5sdtvq"
Aug 04 18:45:53.536757 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:45:53.536731    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c3091da-e8ba-4c2c-9d26-f02d4c5835bb-home\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-579b7ffcc5sdtvq\" (UID: \"5c3091da-e8ba-4c2c-9d26-f02d4c5835bb\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-579b7ffcc5sdtvq"
Aug 04 18:45:53.637602 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:45:53.637558    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mmw7\" (UniqueName: \"kubernetes.io/projected/5c3091da-e8ba-4c2c-9d26-f02d4c5835bb-kube-api-access-8mmw7\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-579b7ffcc5sdtvq\" (UID: \"5c3091da-e8ba-4c2c-9d26-f02d4c5835bb\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-579b7ffcc5sdtvq"
Aug 04 18:45:53.637809 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:45:53.637628    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5c3091da-e8ba-4c2c-9d26-f02d4c5835bb-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-579b7ffcc5sdtvq\" (UID: \"5c3091da-e8ba-4c2c-9d26-f02d4c5835bb\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-579b7ffcc5sdtvq"
Aug 04 18:45:53.637809 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:45:53.637745    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c3091da-e8ba-4c2c-9d26-f02d4c5835bb-dshm\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-579b7ffcc5sdtvq\" (UID: \"5c3091da-e8ba-4c2c-9d26-f02d4c5835bb\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-579b7ffcc5sdtvq"
Aug 04 18:45:53.637809 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:45:53.637802    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c3091da-e8ba-4c2c-9d26-f02d4c5835bb-home\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-579b7ffcc5sdtvq\" (UID: \"5c3091da-e8ba-4c2c-9d26-f02d4c5835bb\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-579b7ffcc5sdtvq"
Aug 04 18:45:53.638003 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:45:53.637846    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c3091da-e8ba-4c2c-9d26-f02d4c5835bb-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-579b7ffcc5sdtvq\" (UID: \"5c3091da-e8ba-4c2c-9d26-f02d4c5835bb\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-579b7ffcc5sdtvq"
Aug 04 18:45:53.638003 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:45:53.637869    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c3091da-e8ba-4c2c-9d26-f02d4c5835bb-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-579b7ffcc5sdtvq\" (UID: \"5c3091da-e8ba-4c2c-9d26-f02d4c5835bb\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-579b7ffcc5sdtvq"
Aug 04 18:45:53.638003 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:45:53.637888    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c3091da-e8ba-4c2c-9d26-f02d4c5835bb-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-579b7ffcc5sdtvq\" (UID: \"5c3091da-e8ba-4c2c-9d26-f02d4c5835bb\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-579b7ffcc5sdtvq"
Aug 04 18:45:53.638003 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:45:53.637933    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5c3091da-e8ba-4c2c-9d26-f02d4c5835bb-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-579b7ffcc5sdtvq\" (UID: \"5c3091da-e8ba-4c2c-9d26-f02d4c5835bb\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-579b7ffcc5sdtvq"
Aug 04 18:45:53.638216 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:45:53.638172    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c3091da-e8ba-4c2c-9d26-f02d4c5835bb-home\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-579b7ffcc5sdtvq\" (UID: \"5c3091da-e8ba-4c2c-9d26-f02d4c5835bb\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-579b7ffcc5sdtvq"
Aug 04 18:45:53.638301 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:45:53.638267    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5c3091da-e8ba-4c2c-9d26-f02d4c5835bb-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-579b7ffcc5sdtvq\" (UID: \"5c3091da-e8ba-4c2c-9d26-f02d4c5835bb\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-579b7ffcc5sdtvq"
Aug 04 18:45:53.638364 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:45:53.638313    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c3091da-e8ba-4c2c-9d26-f02d4c5835bb-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-579b7ffcc5sdtvq\" (UID: \"5c3091da-e8ba-4c2c-9d26-f02d4c5835bb\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-579b7ffcc5sdtvq"
Aug 04 18:45:53.638364 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:45:53.638333    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c3091da-e8ba-4c2c-9d26-f02d4c5835bb-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-579b7ffcc5sdtvq\" (UID: \"5c3091da-e8ba-4c2c-9d26-f02d4c5835bb\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-579b7ffcc5sdtvq"
Aug 04 18:45:53.638525 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:45:53.638504    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5c3091da-e8ba-4c2c-9d26-f02d4c5835bb-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-579b7ffcc5sdtvq\" (UID: \"5c3091da-e8ba-4c2c-9d26-f02d4c5835bb\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-579b7ffcc5sdtvq"
Aug 04 18:45:53.640113 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:45:53.640095    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c3091da-e8ba-4c2c-9d26-f02d4c5835bb-dshm\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-579b7ffcc5sdtvq\" (UID: \"5c3091da-e8ba-4c2c-9d26-f02d4c5835bb\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-579b7ffcc5sdtvq"
Aug 04 18:45:53.640392 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:45:53.640375    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c3091da-e8ba-4c2c-9d26-f02d4c5835bb-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-579b7ffcc5sdtvq\" (UID: \"5c3091da-e8ba-4c2c-9d26-f02d4c5835bb\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-579b7ffcc5sdtvq"
Aug 04 18:45:53.646742 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:45:53.646722    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mmw7\" (UniqueName: \"kubernetes.io/projected/5c3091da-e8ba-4c2c-9d26-f02d4c5835bb-kube-api-access-8mmw7\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-579b7ffcc5sdtvq\" (UID: \"5c3091da-e8ba-4c2c-9d26-f02d4c5835bb\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-579b7ffcc5sdtvq"
Aug 04 18:45:53.739239 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:45:53.739193    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-579b7ffcc5sdtvq"
Aug 04 18:45:53.883670 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:45:53.883639    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-579b7ffcc5sdtvq"]
Aug 04 18:45:53.885867 ip-10-0-135-124 kubenswrapper[2583]: W0804 18:45:53.885841    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c3091da_e8ba_4c2c_9d26_f02d4c5835bb.slice/crio-f3b988cb90a16235b77a70ba373bd24402bb55162f96bafd70292c51e884eaaf WatchSource:0}: Error finding container f3b988cb90a16235b77a70ba373bd24402bb55162f96bafd70292c51e884eaaf: Status 404 returned error can't find the container with id f3b988cb90a16235b77a70ba373bd24402bb55162f96bafd70292c51e884eaaf
Aug 04 18:45:54.220310 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:45:54.220271    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-579b7ffcc5sdtvq" event={"ID":"5c3091da-e8ba-4c2c-9d26-f02d4c5835bb","Type":"ContainerStarted","Data":"8e2969c4a3ca9b2137d8c987650c3ccaae2a11d6d11fbf0b8b8f6c8f74ed48c5"}
Aug 04 18:45:54.220498 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:45:54.220332    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-579b7ffcc5sdtvq" event={"ID":"5c3091da-e8ba-4c2c-9d26-f02d4c5835bb","Type":"ContainerStarted","Data":"f3b988cb90a16235b77a70ba373bd24402bb55162f96bafd70292c51e884eaaf"}
Aug 04 18:45:56.233007 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:45:56.232966    2583 generic.go:356] "Generic (PLEG): container finished" podID="5c3091da-e8ba-4c2c-9d26-f02d4c5835bb" containerID="8e2969c4a3ca9b2137d8c987650c3ccaae2a11d6d11fbf0b8b8f6c8f74ed48c5" exitCode=0
Aug 04 18:45:56.233436 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:45:56.233033    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-579b7ffcc5sdtvq" event={"ID":"5c3091da-e8ba-4c2c-9d26-f02d4c5835bb","Type":"ContainerDied","Data":"8e2969c4a3ca9b2137d8c987650c3ccaae2a11d6d11fbf0b8b8f6c8f74ed48c5"}
Aug 04 18:45:57.241546 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:45:57.241509    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-579b7ffcc5sdtvq" event={"ID":"5c3091da-e8ba-4c2c-9d26-f02d4c5835bb","Type":"ContainerStarted","Data":"406a9563fcd74695bbb101cc2d22472b7e95c14723344eb12dabee91ba953874"}
Aug 04 18:45:57.267840 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:45:57.267787    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-579b7ffcc5sdtvq" podStartSLOduration=2.0965291600000002 podStartE2EDuration="4.267773161s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.171244001s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:45:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:45:57.264407878 +0000 UTC m=+2770.845626455" watchObservedRunningTime="2026-08-04 18:45:57.267773161 +0000 UTC m=+2770.848991739"
Aug 04 18:45:57.739415 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:45:57.739378    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-579b7ffcc5sdtvq"
Aug 04 18:46:03.739634 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:46:03.739584    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-579b7ffcc5sdtvq"
Aug 04 18:46:03.741488 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:46:03.741434    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-579b7ffcc5sdtvq" podUID="5c3091da-e8ba-4c2c-9d26-f02d4c5835bb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": dial tcp 10.132.0.80:8000: connect: connection refused"
Aug 04 18:46:13.740302 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:46:13.740215    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-579b7ffcc5sdtvq" podUID="5c3091da-e8ba-4c2c-9d26-f02d4c5835bb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": dial tcp 10.132.0.80:8000: connect: connection refused"
Aug 04 18:46:23.740378 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:46:23.740334    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-579b7ffcc5sdtvq" podUID="5c3091da-e8ba-4c2c-9d26-f02d4c5835bb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": dial tcp 10.132.0.80:8000: connect: connection refused"
Aug 04 18:46:33.739830 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:46:33.739784    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-579b7ffcc5sdtvq" podUID="5c3091da-e8ba-4c2c-9d26-f02d4c5835bb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": dial tcp 10.132.0.80:8000: connect: connection refused"
Aug 04 18:46:43.740486 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:46:43.740426    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-579b7ffcc5sdtvq" podUID="5c3091da-e8ba-4c2c-9d26-f02d4c5835bb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": dial tcp 10.132.0.80:8000: connect: connection refused"
Aug 04 18:46:53.740662 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:46:53.740622    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-579b7ffcc5sdtvq" podUID="5c3091da-e8ba-4c2c-9d26-f02d4c5835bb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": dial tcp 10.132.0.80:8000: connect: connection refused"
Aug 04 18:47:03.740356 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:47:03.740302    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-579b7ffcc5sdtvq" podUID="5c3091da-e8ba-4c2c-9d26-f02d4c5835bb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": dial tcp 10.132.0.80:8000: connect: connection refused"
Aug 04 18:47:13.739910 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:47:13.739865    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-579b7ffcc5sdtvq" podUID="5c3091da-e8ba-4c2c-9d26-f02d4c5835bb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": dial tcp 10.132.0.80:8000: connect: connection refused"
Aug 04 18:47:23.740261 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:47:23.740215    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-579b7ffcc5sdtvq" podUID="5c3091da-e8ba-4c2c-9d26-f02d4c5835bb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": dial tcp 10.132.0.80:8000: connect: connection refused"
Aug 04 18:47:33.739904 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:47:33.739859    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-579b7ffcc5sdtvq" podUID="5c3091da-e8ba-4c2c-9d26-f02d4c5835bb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": dial tcp 10.132.0.80:8000: connect: connection refused"
Aug 04 18:47:43.740750 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:47:43.740702    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-579b7ffcc5sdtvq" podUID="5c3091da-e8ba-4c2c-9d26-f02d4c5835bb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": dial tcp 10.132.0.80:8000: connect: connection refused"
Aug 04 18:47:53.749829 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:47:53.749792    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-579b7ffcc5sdtvq"
Aug 04 18:47:53.758235 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:47:53.758202    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-579b7ffcc5sdtvq"
Aug 04 18:47:58.951774 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:47:58.951733    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-579b7ffcc5sdtvq"]
Aug 04 18:48:13.974927 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:48:13.974887    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-579b7ffcc5sdtvq" podUID="5c3091da-e8ba-4c2c-9d26-f02d4c5835bb" containerName="main" containerID="cri-o://406a9563fcd74695bbb101cc2d22472b7e95c14723344eb12dabee91ba953874" gracePeriod=45
Aug 04 18:48:14.731201 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:48:14.731173    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-579b7ffcc5sdtvq"
Aug 04 18:48:14.761788 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:48:14.761703    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5c3091da-e8ba-4c2c-9d26-f02d4c5835bb-cabundle-cert\") pod \"5c3091da-e8ba-4c2c-9d26-f02d4c5835bb\" (UID: \"5c3091da-e8ba-4c2c-9d26-f02d4c5835bb\") "
Aug 04 18:48:14.761788 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:48:14.761737    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c3091da-e8ba-4c2c-9d26-f02d4c5835bb-dshm\") pod \"5c3091da-e8ba-4c2c-9d26-f02d4c5835bb\" (UID: \"5c3091da-e8ba-4c2c-9d26-f02d4c5835bb\") "
Aug 04 18:48:14.761788 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:48:14.761775    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8mmw7\" (UniqueName: \"kubernetes.io/projected/5c3091da-e8ba-4c2c-9d26-f02d4c5835bb-kube-api-access-8mmw7\") pod \"5c3091da-e8ba-4c2c-9d26-f02d4c5835bb\" (UID: \"5c3091da-e8ba-4c2c-9d26-f02d4c5835bb\") "
Aug 04 18:48:14.762264 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:48:14.761797    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c3091da-e8ba-4c2c-9d26-f02d4c5835bb-tls-certs\") pod \"5c3091da-e8ba-4c2c-9d26-f02d4c5835bb\" (UID: \"5c3091da-e8ba-4c2c-9d26-f02d4c5835bb\") "
Aug 04 18:48:14.762264 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:48:14.761835    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c3091da-e8ba-4c2c-9d26-f02d4c5835bb-home\") pod \"5c3091da-e8ba-4c2c-9d26-f02d4c5835bb\" (UID: \"5c3091da-e8ba-4c2c-9d26-f02d4c5835bb\") "
Aug 04 18:48:14.762264 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:48:14.761868    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c3091da-e8ba-4c2c-9d26-f02d4c5835bb-model-cache\") pod \"5c3091da-e8ba-4c2c-9d26-f02d4c5835bb\" (UID: \"5c3091da-e8ba-4c2c-9d26-f02d4c5835bb\") "
Aug 04 18:48:14.762264 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:48:14.761888    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c3091da-e8ba-4c2c-9d26-f02d4c5835bb-tmp-dir\") pod \"5c3091da-e8ba-4c2c-9d26-f02d4c5835bb\" (UID: \"5c3091da-e8ba-4c2c-9d26-f02d4c5835bb\") "
Aug 04 18:48:14.762264 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:48:14.761957    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5c3091da-e8ba-4c2c-9d26-f02d4c5835bb-kserve-provision-location\") pod \"5c3091da-e8ba-4c2c-9d26-f02d4c5835bb\" (UID: \"5c3091da-e8ba-4c2c-9d26-f02d4c5835bb\") "
Aug 04 18:48:14.762649 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:48:14.762450    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c3091da-e8ba-4c2c-9d26-f02d4c5835bb-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "5c3091da-e8ba-4c2c-9d26-f02d4c5835bb" (UID: "5c3091da-e8ba-4c2c-9d26-f02d4c5835bb"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:48:14.762934 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:48:14.762903    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c3091da-e8ba-4c2c-9d26-f02d4c5835bb-model-cache" (OuterVolumeSpecName: "model-cache") pod "5c3091da-e8ba-4c2c-9d26-f02d4c5835bb" (UID: "5c3091da-e8ba-4c2c-9d26-f02d4c5835bb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:48:14.763352 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:48:14.763304    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c3091da-e8ba-4c2c-9d26-f02d4c5835bb-home" (OuterVolumeSpecName: "home") pod "5c3091da-e8ba-4c2c-9d26-f02d4c5835bb" (UID: "5c3091da-e8ba-4c2c-9d26-f02d4c5835bb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:48:14.765007 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:48:14.764962    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c3091da-e8ba-4c2c-9d26-f02d4c5835bb-kube-api-access-8mmw7" (OuterVolumeSpecName: "kube-api-access-8mmw7") pod "5c3091da-e8ba-4c2c-9d26-f02d4c5835bb" (UID: "5c3091da-e8ba-4c2c-9d26-f02d4c5835bb"). InnerVolumeSpecName "kube-api-access-8mmw7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:48:14.765449 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:48:14.765420    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c3091da-e8ba-4c2c-9d26-f02d4c5835bb-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5c3091da-e8ba-4c2c-9d26-f02d4c5835bb" (UID: "5c3091da-e8ba-4c2c-9d26-f02d4c5835bb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:48:14.765604 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:48:14.765455    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c3091da-e8ba-4c2c-9d26-f02d4c5835bb-dshm" (OuterVolumeSpecName: "dshm") pod "5c3091da-e8ba-4c2c-9d26-f02d4c5835bb" (UID: "5c3091da-e8ba-4c2c-9d26-f02d4c5835bb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:48:14.776496 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:48:14.776451    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c3091da-e8ba-4c2c-9d26-f02d4c5835bb-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5c3091da-e8ba-4c2c-9d26-f02d4c5835bb" (UID: "5c3091da-e8ba-4c2c-9d26-f02d4c5835bb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:48:14.787419 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:48:14.787394    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c3091da-e8ba-4c2c-9d26-f02d4c5835bb-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5c3091da-e8ba-4c2c-9d26-f02d4c5835bb" (UID: "5c3091da-e8ba-4c2c-9d26-f02d4c5835bb"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:48:14.863424 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:48:14.863389    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5c3091da-e8ba-4c2c-9d26-f02d4c5835bb-kserve-provision-location\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:48:14.863424 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:48:14.863421    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5c3091da-e8ba-4c2c-9d26-f02d4c5835bb-cabundle-cert\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:48:14.863619 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:48:14.863433    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c3091da-e8ba-4c2c-9d26-f02d4c5835bb-dshm\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:48:14.863619 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:48:14.863441    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8mmw7\" (UniqueName: \"kubernetes.io/projected/5c3091da-e8ba-4c2c-9d26-f02d4c5835bb-kube-api-access-8mmw7\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:48:14.863619 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:48:14.863450    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c3091da-e8ba-4c2c-9d26-f02d4c5835bb-tls-certs\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:48:14.863619 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:48:14.863458    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c3091da-e8ba-4c2c-9d26-f02d4c5835bb-home\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:48:14.863619 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:48:14.863485    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c3091da-e8ba-4c2c-9d26-f02d4c5835bb-model-cache\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:48:14.863619 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:48:14.863520    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c3091da-e8ba-4c2c-9d26-f02d4c5835bb-tmp-dir\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:48:15.012707 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:48:15.012617    2583 generic.go:356] "Generic (PLEG): container finished" podID="5c3091da-e8ba-4c2c-9d26-f02d4c5835bb" containerID="406a9563fcd74695bbb101cc2d22472b7e95c14723344eb12dabee91ba953874" exitCode=0
Aug 04 18:48:15.012707 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:48:15.012677    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-579b7ffcc5sdtvq"
Aug 04 18:48:15.013146 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:48:15.012704    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-579b7ffcc5sdtvq" event={"ID":"5c3091da-e8ba-4c2c-9d26-f02d4c5835bb","Type":"ContainerDied","Data":"406a9563fcd74695bbb101cc2d22472b7e95c14723344eb12dabee91ba953874"}
Aug 04 18:48:15.013146 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:48:15.012765    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-579b7ffcc5sdtvq" event={"ID":"5c3091da-e8ba-4c2c-9d26-f02d4c5835bb","Type":"ContainerDied","Data":"f3b988cb90a16235b77a70ba373bd24402bb55162f96bafd70292c51e884eaaf"}
Aug 04 18:48:15.013146 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:48:15.012787    2583 scope.go:117] "RemoveContainer" containerID="406a9563fcd74695bbb101cc2d22472b7e95c14723344eb12dabee91ba953874"
Aug 04 18:48:15.021536 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:48:15.021513    2583 scope.go:117] "RemoveContainer" containerID="8e2969c4a3ca9b2137d8c987650c3ccaae2a11d6d11fbf0b8b8f6c8f74ed48c5"
Aug 04 18:48:15.029258 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:48:15.029237    2583 scope.go:117] "RemoveContainer" containerID="406a9563fcd74695bbb101cc2d22472b7e95c14723344eb12dabee91ba953874"
Aug 04 18:48:15.029521 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:48:15.029500    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"406a9563fcd74695bbb101cc2d22472b7e95c14723344eb12dabee91ba953874\": container with ID starting with 406a9563fcd74695bbb101cc2d22472b7e95c14723344eb12dabee91ba953874 not found: ID does not exist" containerID="406a9563fcd74695bbb101cc2d22472b7e95c14723344eb12dabee91ba953874"
Aug 04 18:48:15.029587 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:48:15.029529    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"406a9563fcd74695bbb101cc2d22472b7e95c14723344eb12dabee91ba953874"} err="failed to get container status \"406a9563fcd74695bbb101cc2d22472b7e95c14723344eb12dabee91ba953874\": rpc error: code = NotFound desc = could not find container \"406a9563fcd74695bbb101cc2d22472b7e95c14723344eb12dabee91ba953874\": container with ID starting with 406a9563fcd74695bbb101cc2d22472b7e95c14723344eb12dabee91ba953874 not found: ID does not exist"
Aug 04 18:48:15.029587 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:48:15.029546    2583 scope.go:117] "RemoveContainer" containerID="8e2969c4a3ca9b2137d8c987650c3ccaae2a11d6d11fbf0b8b8f6c8f74ed48c5"
Aug 04 18:48:15.029767 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:48:15.029749    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e2969c4a3ca9b2137d8c987650c3ccaae2a11d6d11fbf0b8b8f6c8f74ed48c5\": container with ID starting with 8e2969c4a3ca9b2137d8c987650c3ccaae2a11d6d11fbf0b8b8f6c8f74ed48c5 not found: ID does not exist" containerID="8e2969c4a3ca9b2137d8c987650c3ccaae2a11d6d11fbf0b8b8f6c8f74ed48c5"
Aug 04 18:48:15.029806 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:48:15.029773    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e2969c4a3ca9b2137d8c987650c3ccaae2a11d6d11fbf0b8b8f6c8f74ed48c5"} err="failed to get container status \"8e2969c4a3ca9b2137d8c987650c3ccaae2a11d6d11fbf0b8b8f6c8f74ed48c5\": rpc error: code = NotFound desc = could not find container \"8e2969c4a3ca9b2137d8c987650c3ccaae2a11d6d11fbf0b8b8f6c8f74ed48c5\": container with ID starting with 8e2969c4a3ca9b2137d8c987650c3ccaae2a11d6d11fbf0b8b8f6c8f74ed48c5 not found: ID does not exist"
Aug 04 18:48:15.034150 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:48:15.034126    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-579b7ffcc5sdtvq"]
Aug 04 18:48:15.038220 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:48:15.038199    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-579b7ffcc5sdtvq"]
Aug 04 18:48:16.942866 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:48:16.942831    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5c3091da-e8ba-4c2c-9d26-f02d4c5835bb" path="/var/lib/kubelet/pods/5c3091da-e8ba-4c2c-9d26-f02d4c5835bb/volumes"
Aug 04 18:48:46.399572 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:48:46.399536    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-845b9884cd-kw7qw"]
Aug 04 18:48:46.400171 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:48:46.400068    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c3091da-e8ba-4c2c-9d26-f02d4c5835bb" containerName="storage-initializer"
Aug 04 18:48:46.400171 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:48:46.400089    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c3091da-e8ba-4c2c-9d26-f02d4c5835bb" containerName="storage-initializer"
Aug 04 18:48:46.400171 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:48:46.400108    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c3091da-e8ba-4c2c-9d26-f02d4c5835bb" containerName="main"
Aug 04 18:48:46.400171 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:48:46.400117    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c3091da-e8ba-4c2c-9d26-f02d4c5835bb" containerName="main"
Aug 04 18:48:46.400424 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:48:46.400245    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="5c3091da-e8ba-4c2c-9d26-f02d4c5835bb" containerName="main"
Aug 04 18:48:46.403581 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:48:46.403526    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-845b9884cd-kw7qw"
Aug 04 18:48:46.406858 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:48:46.406837    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"kube-root-ca.crt\""
Aug 04 18:48:46.406983 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:48:46.406837    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"odh-kserve-custom-ca-bundle\""
Aug 04 18:48:46.406983 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:48:46.406887    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"custom-route-timeout-test-kserve-self-signed-certs\""
Aug 04 18:48:46.406983 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:48:46.406966    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"seaweedfs-s3-creds\""
Aug 04 18:48:46.407164 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:48:46.406992    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"default-dockercfg-m6frn\""
Aug 04 18:48:46.407310 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:48:46.407293    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"openshift-service-ca.crt\""
Aug 04 18:48:46.416784 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:48:46.416760    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-845b9884cd-kw7qw"]
Aug 04 18:48:46.537861 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:48:46.537821    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b02b522f-0b35-4355-9492-57a78506afcf-tmp-dir\") pod \"custom-route-timeout-test-kserve-845b9884cd-kw7qw\" (UID: \"b02b522f-0b35-4355-9492-57a78506afcf\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-845b9884cd-kw7qw"
Aug 04 18:48:46.538032 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:48:46.537875    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b02b522f-0b35-4355-9492-57a78506afcf-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-845b9884cd-kw7qw\" (UID: \"b02b522f-0b35-4355-9492-57a78506afcf\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-845b9884cd-kw7qw"
Aug 04 18:48:46.538032 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:48:46.537927    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b02b522f-0b35-4355-9492-57a78506afcf-cabundle-cert\") pod \"custom-route-timeout-test-kserve-845b9884cd-kw7qw\" (UID: \"b02b522f-0b35-4355-9492-57a78506afcf\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-845b9884cd-kw7qw"
Aug 04 18:48:46.538032 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:48:46.537992    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b02b522f-0b35-4355-9492-57a78506afcf-tls-certs\") pod \"custom-route-timeout-test-kserve-845b9884cd-kw7qw\" (UID: \"b02b522f-0b35-4355-9492-57a78506afcf\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-845b9884cd-kw7qw"
Aug 04 18:48:46.538032 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:48:46.538027    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dpxl2\" (UniqueName: \"kubernetes.io/projected/b02b522f-0b35-4355-9492-57a78506afcf-kube-api-access-dpxl2\") pod \"custom-route-timeout-test-kserve-845b9884cd-kw7qw\" (UID: \"b02b522f-0b35-4355-9492-57a78506afcf\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-845b9884cd-kw7qw"
Aug 04 18:48:46.538177 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:48:46.538047    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b02b522f-0b35-4355-9492-57a78506afcf-home\") pod \"custom-route-timeout-test-kserve-845b9884cd-kw7qw\" (UID: \"b02b522f-0b35-4355-9492-57a78506afcf\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-845b9884cd-kw7qw"
Aug 04 18:48:46.538177 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:48:46.538061    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b02b522f-0b35-4355-9492-57a78506afcf-dshm\") pod \"custom-route-timeout-test-kserve-845b9884cd-kw7qw\" (UID: \"b02b522f-0b35-4355-9492-57a78506afcf\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-845b9884cd-kw7qw"
Aug 04 18:48:46.538177 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:48:46.538078    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b02b522f-0b35-4355-9492-57a78506afcf-model-cache\") pod \"custom-route-timeout-test-kserve-845b9884cd-kw7qw\" (UID: \"b02b522f-0b35-4355-9492-57a78506afcf\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-845b9884cd-kw7qw"
Aug 04 18:48:46.639226 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:48:46.639185    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b02b522f-0b35-4355-9492-57a78506afcf-tmp-dir\") pod \"custom-route-timeout-test-kserve-845b9884cd-kw7qw\" (UID: \"b02b522f-0b35-4355-9492-57a78506afcf\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-845b9884cd-kw7qw"
Aug 04 18:48:46.639391 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:48:46.639238    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b02b522f-0b35-4355-9492-57a78506afcf-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-845b9884cd-kw7qw\" (UID: \"b02b522f-0b35-4355-9492-57a78506afcf\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-845b9884cd-kw7qw"
Aug 04 18:48:46.639391 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:48:46.639356    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b02b522f-0b35-4355-9492-57a78506afcf-cabundle-cert\") pod \"custom-route-timeout-test-kserve-845b9884cd-kw7qw\" (UID: \"b02b522f-0b35-4355-9492-57a78506afcf\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-845b9884cd-kw7qw"
Aug 04 18:48:46.639483 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:48:46.639442    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b02b522f-0b35-4355-9492-57a78506afcf-tls-certs\") pod \"custom-route-timeout-test-kserve-845b9884cd-kw7qw\" (UID: \"b02b522f-0b35-4355-9492-57a78506afcf\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-845b9884cd-kw7qw"
Aug 04 18:48:46.639540 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:48:46.639514    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpxl2\" (UniqueName: \"kubernetes.io/projected/b02b522f-0b35-4355-9492-57a78506afcf-kube-api-access-dpxl2\") pod \"custom-route-timeout-test-kserve-845b9884cd-kw7qw\" (UID: \"b02b522f-0b35-4355-9492-57a78506afcf\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-845b9884cd-kw7qw"
Aug 04 18:48:46.639596 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:48:46.639543    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b02b522f-0b35-4355-9492-57a78506afcf-home\") pod \"custom-route-timeout-test-kserve-845b9884cd-kw7qw\" (UID: \"b02b522f-0b35-4355-9492-57a78506afcf\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-845b9884cd-kw7qw"
Aug 04 18:48:46.639596 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:48:46.639568    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b02b522f-0b35-4355-9492-57a78506afcf-dshm\") pod \"custom-route-timeout-test-kserve-845b9884cd-kw7qw\" (UID: \"b02b522f-0b35-4355-9492-57a78506afcf\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-845b9884cd-kw7qw"
Aug 04 18:48:46.639694 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:48:46.639597    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b02b522f-0b35-4355-9492-57a78506afcf-model-cache\") pod \"custom-route-timeout-test-kserve-845b9884cd-kw7qw\" (UID: \"b02b522f-0b35-4355-9492-57a78506afcf\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-845b9884cd-kw7qw"
Aug 04 18:48:46.639744 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:48:46.639728    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b02b522f-0b35-4355-9492-57a78506afcf-tmp-dir\") pod \"custom-route-timeout-test-kserve-845b9884cd-kw7qw\" (UID: \"b02b522f-0b35-4355-9492-57a78506afcf\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-845b9884cd-kw7qw"
Aug 04 18:48:46.639845 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:48:46.639815    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b02b522f-0b35-4355-9492-57a78506afcf-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-845b9884cd-kw7qw\" (UID: \"b02b522f-0b35-4355-9492-57a78506afcf\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-845b9884cd-kw7qw"
Aug 04 18:48:46.640013 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:48:46.639981    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b02b522f-0b35-4355-9492-57a78506afcf-home\") pod \"custom-route-timeout-test-kserve-845b9884cd-kw7qw\" (UID: \"b02b522f-0b35-4355-9492-57a78506afcf\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-845b9884cd-kw7qw"
Aug 04 18:48:46.640110 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:48:46.640069    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b02b522f-0b35-4355-9492-57a78506afcf-model-cache\") pod \"custom-route-timeout-test-kserve-845b9884cd-kw7qw\" (UID: \"b02b522f-0b35-4355-9492-57a78506afcf\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-845b9884cd-kw7qw"
Aug 04 18:48:46.640229 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:48:46.640209    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b02b522f-0b35-4355-9492-57a78506afcf-cabundle-cert\") pod \"custom-route-timeout-test-kserve-845b9884cd-kw7qw\" (UID: \"b02b522f-0b35-4355-9492-57a78506afcf\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-845b9884cd-kw7qw"
Aug 04 18:48:46.641935 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:48:46.641914    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b02b522f-0b35-4355-9492-57a78506afcf-dshm\") pod \"custom-route-timeout-test-kserve-845b9884cd-kw7qw\" (UID: \"b02b522f-0b35-4355-9492-57a78506afcf\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-845b9884cd-kw7qw"
Aug 04 18:48:46.642188 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:48:46.642167    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b02b522f-0b35-4355-9492-57a78506afcf-tls-certs\") pod \"custom-route-timeout-test-kserve-845b9884cd-kw7qw\" (UID: \"b02b522f-0b35-4355-9492-57a78506afcf\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-845b9884cd-kw7qw"
Aug 04 18:48:46.648481 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:48:46.648443    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dpxl2\" (UniqueName: \"kubernetes.io/projected/b02b522f-0b35-4355-9492-57a78506afcf-kube-api-access-dpxl2\") pod \"custom-route-timeout-test-kserve-845b9884cd-kw7qw\" (UID: \"b02b522f-0b35-4355-9492-57a78506afcf\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-845b9884cd-kw7qw"
Aug 04 18:48:46.714983 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:48:46.714950    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-845b9884cd-kw7qw"
Aug 04 18:48:46.862333 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:48:46.862304    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-845b9884cd-kw7qw"]
Aug 04 18:48:46.863676 ip-10-0-135-124 kubenswrapper[2583]: W0804 18:48:46.863648    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb02b522f_0b35_4355_9492_57a78506afcf.slice/crio-a17685052dccec2b432e8d8bf0def54c618912e0a66c991ddc28b89c4ab76730 WatchSource:0}: Error finding container a17685052dccec2b432e8d8bf0def54c618912e0a66c991ddc28b89c4ab76730: Status 404 returned error can't find the container with id a17685052dccec2b432e8d8bf0def54c618912e0a66c991ddc28b89c4ab76730
Aug 04 18:48:46.868113 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:48:46.868091    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"seaweedfs-s3-creds\""
Aug 04 18:48:47.186031 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:48:47.185991    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-845b9884cd-kw7qw" event={"ID":"b02b522f-0b35-4355-9492-57a78506afcf","Type":"ContainerStarted","Data":"e2ed174504ac4308e0330201d6c8f9df44304cad67c3e1df29116fa02ed0a784"}
Aug 04 18:48:47.186203 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:48:47.186055    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-845b9884cd-kw7qw" event={"ID":"b02b522f-0b35-4355-9492-57a78506afcf","Type":"ContainerStarted","Data":"a17685052dccec2b432e8d8bf0def54c618912e0a66c991ddc28b89c4ab76730"}
Aug 04 18:48:49.199266 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:48:49.199228    2583 generic.go:356] "Generic (PLEG): container finished" podID="b02b522f-0b35-4355-9492-57a78506afcf" containerID="e2ed174504ac4308e0330201d6c8f9df44304cad67c3e1df29116fa02ed0a784" exitCode=0
Aug 04 18:48:49.199748 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:48:49.199308    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-845b9884cd-kw7qw" event={"ID":"b02b522f-0b35-4355-9492-57a78506afcf","Type":"ContainerDied","Data":"e2ed174504ac4308e0330201d6c8f9df44304cad67c3e1df29116fa02ed0a784"}
Aug 04 18:48:50.208691 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:48:50.208654    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-845b9884cd-kw7qw" event={"ID":"b02b522f-0b35-4355-9492-57a78506afcf","Type":"ContainerStarted","Data":"7ee998eaadef36b7fe3d543e91be048ac07277c7d3dac3bd69076a432e860bc0"}
Aug 04 18:48:50.235720 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:48:50.235663    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-845b9884cd-kw7qw" podStartSLOduration=2.109219131 podStartE2EDuration="4.23564529s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.126426159s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:48:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:48:50.234251415 +0000 UTC m=+2943.815469995" watchObservedRunningTime="2026-08-04 18:48:50.23564529 +0000 UTC m=+2943.816863867"
Aug 04 18:48:50.715796 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:48:50.715762    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-845b9884cd-kw7qw"
Aug 04 18:48:56.715489 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:48:56.715417    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-845b9884cd-kw7qw"
Aug 04 18:48:56.717178 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:48:56.717145    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-845b9884cd-kw7qw" podUID="b02b522f-0b35-4355-9492-57a78506afcf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/health\": dial tcp 10.132.0.81:8000: connect: connection refused"
Aug 04 18:49:06.715906 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:49:06.715850    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-845b9884cd-kw7qw" podUID="b02b522f-0b35-4355-9492-57a78506afcf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/health\": dial tcp 10.132.0.81:8000: connect: connection refused"
Aug 04 18:49:16.715968 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:49:16.715864    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-845b9884cd-kw7qw" podUID="b02b522f-0b35-4355-9492-57a78506afcf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/health\": dial tcp 10.132.0.81:8000: connect: connection refused"
Aug 04 18:49:26.716244 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:49:26.716197    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-845b9884cd-kw7qw" podUID="b02b522f-0b35-4355-9492-57a78506afcf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/health\": dial tcp 10.132.0.81:8000: connect: connection refused"
Aug 04 18:49:36.715733 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:49:36.715691    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-845b9884cd-kw7qw" podUID="b02b522f-0b35-4355-9492-57a78506afcf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/health\": dial tcp 10.132.0.81:8000: connect: connection refused"
Aug 04 18:49:46.716093 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:49:46.716045    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-845b9884cd-kw7qw" podUID="b02b522f-0b35-4355-9492-57a78506afcf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/health\": dial tcp 10.132.0.81:8000: connect: connection refused"
Aug 04 18:49:47.178950 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:49:47.178795    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-wskfb_6157c2d3-2fd0-4544-88ad-434ac75f8c04/console-operator/1.log"
Aug 04 18:49:47.190648 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:49:47.190622    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-wskfb_6157c2d3-2fd0-4544-88ad-434ac75f8c04/console-operator/1.log"
Aug 04 18:49:56.715640 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:49:56.715596    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-845b9884cd-kw7qw" podUID="b02b522f-0b35-4355-9492-57a78506afcf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/health\": dial tcp 10.132.0.81:8000: connect: connection refused"
Aug 04 18:50:06.716235 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:50:06.716196    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-845b9884cd-kw7qw" podUID="b02b522f-0b35-4355-9492-57a78506afcf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/health\": dial tcp 10.132.0.81:8000: connect: connection refused"
Aug 04 18:50:16.715621 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:50:16.715570    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-845b9884cd-kw7qw" podUID="b02b522f-0b35-4355-9492-57a78506afcf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/health\": dial tcp 10.132.0.81:8000: connect: connection refused"
Aug 04 18:50:26.715964 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:50:26.715910    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-845b9884cd-kw7qw" podUID="b02b522f-0b35-4355-9492-57a78506afcf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/health\": dial tcp 10.132.0.81:8000: connect: connection refused"
Aug 04 18:50:36.715995 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:50:36.715941    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-845b9884cd-kw7qw" podUID="b02b522f-0b35-4355-9492-57a78506afcf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/health\": dial tcp 10.132.0.81:8000: connect: connection refused"
Aug 04 18:50:46.726276 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:50:46.726243    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-845b9884cd-kw7qw"
Aug 04 18:50:46.734030 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:50:46.734008    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-845b9884cd-kw7qw"
Aug 04 18:50:51.261136 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:50:51.261094    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-845b9884cd-kw7qw"]
Aug 04 18:51:06.284366 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:51:06.284283    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-845b9884cd-kw7qw" podUID="b02b522f-0b35-4355-9492-57a78506afcf" containerName="main" containerID="cri-o://7ee998eaadef36b7fe3d543e91be048ac07277c7d3dac3bd69076a432e860bc0" gracePeriod=45
Aug 04 18:51:06.726930 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:51:06.726886    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-845b9884cd-kw7qw" podUID="b02b522f-0b35-4355-9492-57a78506afcf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/health\": dial tcp 10.132.0.81:8000: connect: connection refused"
Aug 04 18:51:06.983485 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:51:06.983383    2583 generic.go:356] "Generic (PLEG): container finished" podID="b02b522f-0b35-4355-9492-57a78506afcf" containerID="7ee998eaadef36b7fe3d543e91be048ac07277c7d3dac3bd69076a432e860bc0" exitCode=0
Aug 04 18:51:06.983651 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:51:06.983489    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-845b9884cd-kw7qw" event={"ID":"b02b522f-0b35-4355-9492-57a78506afcf","Type":"ContainerDied","Data":"7ee998eaadef36b7fe3d543e91be048ac07277c7d3dac3bd69076a432e860bc0"}
Aug 04 18:51:07.041167 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:51:07.041142    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-845b9884cd-kw7qw"
Aug 04 18:51:07.164683 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:51:07.164651    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dpxl2\" (UniqueName: \"kubernetes.io/projected/b02b522f-0b35-4355-9492-57a78506afcf-kube-api-access-dpxl2\") pod \"b02b522f-0b35-4355-9492-57a78506afcf\" (UID: \"b02b522f-0b35-4355-9492-57a78506afcf\") "
Aug 04 18:51:07.164856 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:51:07.164703    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b02b522f-0b35-4355-9492-57a78506afcf-dshm\") pod \"b02b522f-0b35-4355-9492-57a78506afcf\" (UID: \"b02b522f-0b35-4355-9492-57a78506afcf\") "
Aug 04 18:51:07.164856 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:51:07.164725    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b02b522f-0b35-4355-9492-57a78506afcf-kserve-provision-location\") pod \"b02b522f-0b35-4355-9492-57a78506afcf\" (UID: \"b02b522f-0b35-4355-9492-57a78506afcf\") "
Aug 04 18:51:07.164856 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:51:07.164752    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b02b522f-0b35-4355-9492-57a78506afcf-cabundle-cert\") pod \"b02b522f-0b35-4355-9492-57a78506afcf\" (UID: \"b02b522f-0b35-4355-9492-57a78506afcf\") "
Aug 04 18:51:07.164856 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:51:07.164800    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b02b522f-0b35-4355-9492-57a78506afcf-tls-certs\") pod \"b02b522f-0b35-4355-9492-57a78506afcf\" (UID: \"b02b522f-0b35-4355-9492-57a78506afcf\") "
Aug 04 18:51:07.164856 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:51:07.164849    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b02b522f-0b35-4355-9492-57a78506afcf-tmp-dir\") pod \"b02b522f-0b35-4355-9492-57a78506afcf\" (UID: \"b02b522f-0b35-4355-9492-57a78506afcf\") "
Aug 04 18:51:07.165142 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:51:07.164980    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b02b522f-0b35-4355-9492-57a78506afcf-home\") pod \"b02b522f-0b35-4355-9492-57a78506afcf\" (UID: \"b02b522f-0b35-4355-9492-57a78506afcf\") "
Aug 04 18:51:07.165142 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:51:07.165017    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b02b522f-0b35-4355-9492-57a78506afcf-model-cache\") pod \"b02b522f-0b35-4355-9492-57a78506afcf\" (UID: \"b02b522f-0b35-4355-9492-57a78506afcf\") "
Aug 04 18:51:07.165254 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:51:07.165168    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b02b522f-0b35-4355-9492-57a78506afcf-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "b02b522f-0b35-4355-9492-57a78506afcf" (UID: "b02b522f-0b35-4355-9492-57a78506afcf"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:51:07.165416 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:51:07.165372    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b02b522f-0b35-4355-9492-57a78506afcf-model-cache" (OuterVolumeSpecName: "model-cache") pod "b02b522f-0b35-4355-9492-57a78506afcf" (UID: "b02b522f-0b35-4355-9492-57a78506afcf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:51:07.165516 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:51:07.165423    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b02b522f-0b35-4355-9492-57a78506afcf-cabundle-cert\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:51:07.165897 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:51:07.165858    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b02b522f-0b35-4355-9492-57a78506afcf-home" (OuterVolumeSpecName: "home") pod "b02b522f-0b35-4355-9492-57a78506afcf" (UID: "b02b522f-0b35-4355-9492-57a78506afcf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:51:07.167455 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:51:07.167422    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b02b522f-0b35-4355-9492-57a78506afcf-kube-api-access-dpxl2" (OuterVolumeSpecName: "kube-api-access-dpxl2") pod "b02b522f-0b35-4355-9492-57a78506afcf" (UID: "b02b522f-0b35-4355-9492-57a78506afcf"). InnerVolumeSpecName "kube-api-access-dpxl2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:51:07.167767 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:51:07.167485    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b02b522f-0b35-4355-9492-57a78506afcf-dshm" (OuterVolumeSpecName: "dshm") pod "b02b522f-0b35-4355-9492-57a78506afcf" (UID: "b02b522f-0b35-4355-9492-57a78506afcf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:51:07.168172 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:51:07.168145    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b02b522f-0b35-4355-9492-57a78506afcf-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b02b522f-0b35-4355-9492-57a78506afcf" (UID: "b02b522f-0b35-4355-9492-57a78506afcf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:51:07.177506 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:51:07.177483    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b02b522f-0b35-4355-9492-57a78506afcf-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b02b522f-0b35-4355-9492-57a78506afcf" (UID: "b02b522f-0b35-4355-9492-57a78506afcf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:51:07.196139 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:51:07.196117    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b02b522f-0b35-4355-9492-57a78506afcf-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b02b522f-0b35-4355-9492-57a78506afcf" (UID: "b02b522f-0b35-4355-9492-57a78506afcf"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:51:07.266569 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:51:07.266505    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dpxl2\" (UniqueName: \"kubernetes.io/projected/b02b522f-0b35-4355-9492-57a78506afcf-kube-api-access-dpxl2\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:51:07.266569 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:51:07.266531    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b02b522f-0b35-4355-9492-57a78506afcf-dshm\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:51:07.266569 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:51:07.266545    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b02b522f-0b35-4355-9492-57a78506afcf-kserve-provision-location\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:51:07.266569 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:51:07.266559    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b02b522f-0b35-4355-9492-57a78506afcf-tls-certs\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:51:07.266768 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:51:07.266572    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b02b522f-0b35-4355-9492-57a78506afcf-tmp-dir\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:51:07.266768 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:51:07.266583    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b02b522f-0b35-4355-9492-57a78506afcf-home\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:51:07.266768 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:51:07.266594    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b02b522f-0b35-4355-9492-57a78506afcf-model-cache\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:51:07.990001 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:51:07.989966    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-845b9884cd-kw7qw" event={"ID":"b02b522f-0b35-4355-9492-57a78506afcf","Type":"ContainerDied","Data":"a17685052dccec2b432e8d8bf0def54c618912e0a66c991ddc28b89c4ab76730"}
Aug 04 18:51:07.990001 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:51:07.989986    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-845b9884cd-kw7qw"
Aug 04 18:51:07.990506 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:51:07.990020    2583 scope.go:117] "RemoveContainer" containerID="7ee998eaadef36b7fe3d543e91be048ac07277c7d3dac3bd69076a432e860bc0"
Aug 04 18:51:08.002316 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:51:08.002290    2583 scope.go:117] "RemoveContainer" containerID="e2ed174504ac4308e0330201d6c8f9df44304cad67c3e1df29116fa02ed0a784"
Aug 04 18:51:08.015081 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:51:08.015059    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-845b9884cd-kw7qw"]
Aug 04 18:51:08.018399 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:51:08.018375    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-845b9884cd-kw7qw"]
Aug 04 18:51:08.942346 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:51:08.942314    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b02b522f-0b35-4355-9492-57a78506afcf" path="/var/lib/kubelet/pods/b02b522f-0b35-4355-9492-57a78506afcf/volumes"
Aug 04 18:51:20.634359 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:51:20.634319    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6bffc6d99d-lshf2"]
Aug 04 18:51:20.634828 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:51:20.634678    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b02b522f-0b35-4355-9492-57a78506afcf" containerName="main"
Aug 04 18:51:20.634828 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:51:20.634689    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="b02b522f-0b35-4355-9492-57a78506afcf" containerName="main"
Aug 04 18:51:20.634828 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:51:20.634709    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b02b522f-0b35-4355-9492-57a78506afcf" containerName="storage-initializer"
Aug 04 18:51:20.634828 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:51:20.634716    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="b02b522f-0b35-4355-9492-57a78506afcf" containerName="storage-initializer"
Aug 04 18:51:20.634828 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:51:20.634768    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="b02b522f-0b35-4355-9492-57a78506afcf" containerName="main"
Aug 04 18:51:20.639557 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:51:20.639540    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6bffc6d99d-lshf2"
Aug 04 18:51:20.642299 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:51:20.642272    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"odh-kserve-custom-ca-bundle\""
Aug 04 18:51:20.642438 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:51:20.642376    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"seaweedfs-s3-creds\""
Aug 04 18:51:20.642438 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:51:20.642428    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"default-dockercfg-wklt5\""
Aug 04 18:51:20.642612 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:51:20.642521    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"router-with-refs-test-kserve-self-signed-certs\""
Aug 04 18:51:20.642612 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:51:20.642596    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"kube-root-ca.crt\""
Aug 04 18:51:20.642701 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:51:20.642613    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"openshift-service-ca.crt\""
Aug 04 18:51:20.649590 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:51:20.649563    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6bffc6d99d-lshf2"]
Aug 04 18:51:20.779601 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:51:20.779561    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17e55bd6-a19b-40e0-a45a-34d7e50b7dd2-home\") pod \"router-with-refs-test-kserve-6bffc6d99d-lshf2\" (UID: \"17e55bd6-a19b-40e0-a45a-34d7e50b7dd2\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6bffc6d99d-lshf2"
Aug 04 18:51:20.779601 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:51:20.779599    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/17e55bd6-a19b-40e0-a45a-34d7e50b7dd2-kserve-provision-location\") pod \"router-with-refs-test-kserve-6bffc6d99d-lshf2\" (UID: \"17e55bd6-a19b-40e0-a45a-34d7e50b7dd2\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6bffc6d99d-lshf2"
Aug 04 18:51:20.779877 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:51:20.779625    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17e55bd6-a19b-40e0-a45a-34d7e50b7dd2-dshm\") pod \"router-with-refs-test-kserve-6bffc6d99d-lshf2\" (UID: \"17e55bd6-a19b-40e0-a45a-34d7e50b7dd2\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6bffc6d99d-lshf2"
Aug 04 18:51:20.779877 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:51:20.779652    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17e55bd6-a19b-40e0-a45a-34d7e50b7dd2-model-cache\") pod \"router-with-refs-test-kserve-6bffc6d99d-lshf2\" (UID: \"17e55bd6-a19b-40e0-a45a-34d7e50b7dd2\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6bffc6d99d-lshf2"
Aug 04 18:51:20.779877 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:51:20.779756    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfwsn\" (UniqueName: \"kubernetes.io/projected/17e55bd6-a19b-40e0-a45a-34d7e50b7dd2-kube-api-access-tfwsn\") pod \"router-with-refs-test-kserve-6bffc6d99d-lshf2\" (UID: \"17e55bd6-a19b-40e0-a45a-34d7e50b7dd2\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6bffc6d99d-lshf2"
Aug 04 18:51:20.779877 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:51:20.779798    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/17e55bd6-a19b-40e0-a45a-34d7e50b7dd2-cabundle-cert\") pod \"router-with-refs-test-kserve-6bffc6d99d-lshf2\" (UID: \"17e55bd6-a19b-40e0-a45a-34d7e50b7dd2\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6bffc6d99d-lshf2"
Aug 04 18:51:20.779877 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:51:20.779871    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17e55bd6-a19b-40e0-a45a-34d7e50b7dd2-tls-certs\") pod \"router-with-refs-test-kserve-6bffc6d99d-lshf2\" (UID: \"17e55bd6-a19b-40e0-a45a-34d7e50b7dd2\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6bffc6d99d-lshf2"
Aug 04 18:51:20.780181 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:51:20.779904    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17e55bd6-a19b-40e0-a45a-34d7e50b7dd2-tmp-dir\") pod \"router-with-refs-test-kserve-6bffc6d99d-lshf2\" (UID: \"17e55bd6-a19b-40e0-a45a-34d7e50b7dd2\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6bffc6d99d-lshf2"
Aug 04 18:51:20.881204 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:51:20.881161    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17e55bd6-a19b-40e0-a45a-34d7e50b7dd2-tls-certs\") pod \"router-with-refs-test-kserve-6bffc6d99d-lshf2\" (UID: \"17e55bd6-a19b-40e0-a45a-34d7e50b7dd2\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6bffc6d99d-lshf2"
Aug 04 18:51:20.881204 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:51:20.881209    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17e55bd6-a19b-40e0-a45a-34d7e50b7dd2-tmp-dir\") pod \"router-with-refs-test-kserve-6bffc6d99d-lshf2\" (UID: \"17e55bd6-a19b-40e0-a45a-34d7e50b7dd2\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6bffc6d99d-lshf2"
Aug 04 18:51:20.881483 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:51:20.881277    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17e55bd6-a19b-40e0-a45a-34d7e50b7dd2-home\") pod \"router-with-refs-test-kserve-6bffc6d99d-lshf2\" (UID: \"17e55bd6-a19b-40e0-a45a-34d7e50b7dd2\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6bffc6d99d-lshf2"
Aug 04 18:51:20.881483 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:51:20.881302    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/17e55bd6-a19b-40e0-a45a-34d7e50b7dd2-kserve-provision-location\") pod \"router-with-refs-test-kserve-6bffc6d99d-lshf2\" (UID: \"17e55bd6-a19b-40e0-a45a-34d7e50b7dd2\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6bffc6d99d-lshf2"
Aug 04 18:51:20.881483 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:51:20.881332    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17e55bd6-a19b-40e0-a45a-34d7e50b7dd2-dshm\") pod \"router-with-refs-test-kserve-6bffc6d99d-lshf2\" (UID: \"17e55bd6-a19b-40e0-a45a-34d7e50b7dd2\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6bffc6d99d-lshf2"
Aug 04 18:51:20.881483 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:51:20.881354    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17e55bd6-a19b-40e0-a45a-34d7e50b7dd2-model-cache\") pod \"router-with-refs-test-kserve-6bffc6d99d-lshf2\" (UID: \"17e55bd6-a19b-40e0-a45a-34d7e50b7dd2\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6bffc6d99d-lshf2"
Aug 04 18:51:20.881679 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:51:20.881513    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfwsn\" (UniqueName: \"kubernetes.io/projected/17e55bd6-a19b-40e0-a45a-34d7e50b7dd2-kube-api-access-tfwsn\") pod \"router-with-refs-test-kserve-6bffc6d99d-lshf2\" (UID: \"17e55bd6-a19b-40e0-a45a-34d7e50b7dd2\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6bffc6d99d-lshf2"
Aug 04 18:51:20.881679 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:51:20.881553    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/17e55bd6-a19b-40e0-a45a-34d7e50b7dd2-cabundle-cert\") pod \"router-with-refs-test-kserve-6bffc6d99d-lshf2\" (UID: \"17e55bd6-a19b-40e0-a45a-34d7e50b7dd2\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6bffc6d99d-lshf2"
Aug 04 18:51:20.881798 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:51:20.881755    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17e55bd6-a19b-40e0-a45a-34d7e50b7dd2-home\") pod \"router-with-refs-test-kserve-6bffc6d99d-lshf2\" (UID: \"17e55bd6-a19b-40e0-a45a-34d7e50b7dd2\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6bffc6d99d-lshf2"
Aug 04 18:51:20.881858 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:51:20.881801    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17e55bd6-a19b-40e0-a45a-34d7e50b7dd2-tmp-dir\") pod \"router-with-refs-test-kserve-6bffc6d99d-lshf2\" (UID: \"17e55bd6-a19b-40e0-a45a-34d7e50b7dd2\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6bffc6d99d-lshf2"
Aug 04 18:51:20.881858 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:51:20.881810    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/17e55bd6-a19b-40e0-a45a-34d7e50b7dd2-kserve-provision-location\") pod \"router-with-refs-test-kserve-6bffc6d99d-lshf2\" (UID: \"17e55bd6-a19b-40e0-a45a-34d7e50b7dd2\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6bffc6d99d-lshf2"
Aug 04 18:51:20.881966 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:51:20.881878    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17e55bd6-a19b-40e0-a45a-34d7e50b7dd2-model-cache\") pod \"router-with-refs-test-kserve-6bffc6d99d-lshf2\" (UID: \"17e55bd6-a19b-40e0-a45a-34d7e50b7dd2\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6bffc6d99d-lshf2"
Aug 04 18:51:20.882274 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:51:20.882256    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/17e55bd6-a19b-40e0-a45a-34d7e50b7dd2-cabundle-cert\") pod \"router-with-refs-test-kserve-6bffc6d99d-lshf2\" (UID: \"17e55bd6-a19b-40e0-a45a-34d7e50b7dd2\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6bffc6d99d-lshf2"
Aug 04 18:51:20.883867 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:51:20.883840    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17e55bd6-a19b-40e0-a45a-34d7e50b7dd2-dshm\") pod \"router-with-refs-test-kserve-6bffc6d99d-lshf2\" (UID: \"17e55bd6-a19b-40e0-a45a-34d7e50b7dd2\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6bffc6d99d-lshf2"
Aug 04 18:51:20.883951 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:51:20.883934    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17e55bd6-a19b-40e0-a45a-34d7e50b7dd2-tls-certs\") pod \"router-with-refs-test-kserve-6bffc6d99d-lshf2\" (UID: \"17e55bd6-a19b-40e0-a45a-34d7e50b7dd2\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6bffc6d99d-lshf2"
Aug 04 18:51:20.891230 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:51:20.891166    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfwsn\" (UniqueName: \"kubernetes.io/projected/17e55bd6-a19b-40e0-a45a-34d7e50b7dd2-kube-api-access-tfwsn\") pod \"router-with-refs-test-kserve-6bffc6d99d-lshf2\" (UID: \"17e55bd6-a19b-40e0-a45a-34d7e50b7dd2\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6bffc6d99d-lshf2"
Aug 04 18:51:20.951703 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:51:20.951669    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6bffc6d99d-lshf2"
Aug 04 18:51:21.102497 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:51:21.102453    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6bffc6d99d-lshf2"]
Aug 04 18:51:21.104072 ip-10-0-135-124 kubenswrapper[2583]: W0804 18:51:21.104044    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17e55bd6_a19b_40e0_a45a_34d7e50b7dd2.slice/crio-83654777d04c5bafc7b5f462b89d436ffd971da75a6e46f6963f53e52c057e83 WatchSource:0}: Error finding container 83654777d04c5bafc7b5f462b89d436ffd971da75a6e46f6963f53e52c057e83: Status 404 returned error can't find the container with id 83654777d04c5bafc7b5f462b89d436ffd971da75a6e46f6963f53e52c057e83
Aug 04 18:51:22.066821 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:51:22.066777    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6bffc6d99d-lshf2" event={"ID":"17e55bd6-a19b-40e0-a45a-34d7e50b7dd2","Type":"ContainerStarted","Data":"c5c5c030d0e0aa0a705db47e4d0099baf5fab9660c3bb08a40682e1ba863c715"}
Aug 04 18:51:22.067272 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:51:22.066845    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6bffc6d99d-lshf2" event={"ID":"17e55bd6-a19b-40e0-a45a-34d7e50b7dd2","Type":"ContainerStarted","Data":"83654777d04c5bafc7b5f462b89d436ffd971da75a6e46f6963f53e52c057e83"}
Aug 04 18:51:24.080331 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:51:24.080291    2583 generic.go:356] "Generic (PLEG): container finished" podID="17e55bd6-a19b-40e0-a45a-34d7e50b7dd2" containerID="c5c5c030d0e0aa0a705db47e4d0099baf5fab9660c3bb08a40682e1ba863c715" exitCode=0
Aug 04 18:51:24.080749 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:51:24.080365    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6bffc6d99d-lshf2" event={"ID":"17e55bd6-a19b-40e0-a45a-34d7e50b7dd2","Type":"ContainerDied","Data":"c5c5c030d0e0aa0a705db47e4d0099baf5fab9660c3bb08a40682e1ba863c715"}
Aug 04 18:51:25.089100 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:51:25.089060    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6bffc6d99d-lshf2" event={"ID":"17e55bd6-a19b-40e0-a45a-34d7e50b7dd2","Type":"ContainerStarted","Data":"09b2783106af4791581c0a237a7c895b087ca2ea43af77eacb0caefbfde54363"}
Aug 04 18:51:25.116840 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:51:25.116769    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6bffc6d99d-lshf2" podStartSLOduration=2.88229574 podStartE2EDuration="5.116747979s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.234452239s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:51:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:51:25.113548384 +0000 UTC m=+3098.694766957" watchObservedRunningTime="2026-08-04 18:51:25.116747979 +0000 UTC m=+3098.697966558"
Aug 04 18:51:25.952525 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:51:25.952482    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6bffc6d99d-lshf2"
Aug 04 18:51:30.952240 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:51:30.952206    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6bffc6d99d-lshf2"
Aug 04 18:51:30.953808 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:51:30.953780    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6bffc6d99d-lshf2" podUID="17e55bd6-a19b-40e0-a45a-34d7e50b7dd2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/health\": dial tcp 10.132.0.82:8000: connect: connection refused"
Aug 04 18:51:40.952958 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:51:40.952913    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6bffc6d99d-lshf2" podUID="17e55bd6-a19b-40e0-a45a-34d7e50b7dd2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/health\": dial tcp 10.132.0.82:8000: connect: connection refused"
Aug 04 18:51:50.952444 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:51:50.952399    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6bffc6d99d-lshf2" podUID="17e55bd6-a19b-40e0-a45a-34d7e50b7dd2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/health\": dial tcp 10.132.0.82:8000: connect: connection refused"
Aug 04 18:52:00.952478 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:52:00.952422    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6bffc6d99d-lshf2" podUID="17e55bd6-a19b-40e0-a45a-34d7e50b7dd2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/health\": dial tcp 10.132.0.82:8000: connect: connection refused"
Aug 04 18:52:10.952612 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:52:10.952530    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6bffc6d99d-lshf2" podUID="17e55bd6-a19b-40e0-a45a-34d7e50b7dd2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/health\": dial tcp 10.132.0.82:8000: connect: connection refused"
Aug 04 18:52:20.953108 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:52:20.953060    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6bffc6d99d-lshf2" podUID="17e55bd6-a19b-40e0-a45a-34d7e50b7dd2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/health\": dial tcp 10.132.0.82:8000: connect: connection refused"
Aug 04 18:52:30.952230 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:52:30.952186    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6bffc6d99d-lshf2" podUID="17e55bd6-a19b-40e0-a45a-34d7e50b7dd2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/health\": dial tcp 10.132.0.82:8000: connect: connection refused"
Aug 04 18:52:40.952481 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:52:40.952429    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6bffc6d99d-lshf2" podUID="17e55bd6-a19b-40e0-a45a-34d7e50b7dd2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/health\": dial tcp 10.132.0.82:8000: connect: connection refused"
Aug 04 18:52:50.952765 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:52:50.952721    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6bffc6d99d-lshf2" podUID="17e55bd6-a19b-40e0-a45a-34d7e50b7dd2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/health\": dial tcp 10.132.0.82:8000: connect: connection refused"
Aug 04 18:53:00.952300 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:53:00.952251    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6bffc6d99d-lshf2" podUID="17e55bd6-a19b-40e0-a45a-34d7e50b7dd2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/health\": dial tcp 10.132.0.82:8000: connect: connection refused"
Aug 04 18:53:10.952162 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:53:10.952127    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6bffc6d99d-lshf2" podUID="17e55bd6-a19b-40e0-a45a-34d7e50b7dd2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/health\": dial tcp 10.132.0.82:8000: connect: connection refused"
Aug 04 18:53:20.962239 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:53:20.962206    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6bffc6d99d-lshf2"
Aug 04 18:53:20.969968 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:53:20.969934    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6bffc6d99d-lshf2"
Aug 04 18:53:25.386017 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:53:25.385972    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6bffc6d99d-lshf2"]
Aug 04 18:53:40.408871 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:53:40.408766    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6bffc6d99d-lshf2" podUID="17e55bd6-a19b-40e0-a45a-34d7e50b7dd2" containerName="main" containerID="cri-o://09b2783106af4791581c0a237a7c895b087ca2ea43af77eacb0caefbfde54363" gracePeriod=45
Aug 04 18:53:40.963379 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:53:40.963342    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6bffc6d99d-lshf2" podUID="17e55bd6-a19b-40e0-a45a-34d7e50b7dd2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/health\": dial tcp 10.132.0.82:8000: connect: connection refused"
Aug 04 18:53:41.167605 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:53:41.167578    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6bffc6d99d-lshf2"
Aug 04 18:53:41.220538 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:53:41.220437    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17e55bd6-a19b-40e0-a45a-34d7e50b7dd2-model-cache\") pod \"17e55bd6-a19b-40e0-a45a-34d7e50b7dd2\" (UID: \"17e55bd6-a19b-40e0-a45a-34d7e50b7dd2\") "
Aug 04 18:53:41.220538 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:53:41.220489    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17e55bd6-a19b-40e0-a45a-34d7e50b7dd2-tls-certs\") pod \"17e55bd6-a19b-40e0-a45a-34d7e50b7dd2\" (UID: \"17e55bd6-a19b-40e0-a45a-34d7e50b7dd2\") "
Aug 04 18:53:41.220538 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:53:41.220511    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/17e55bd6-a19b-40e0-a45a-34d7e50b7dd2-kserve-provision-location\") pod \"17e55bd6-a19b-40e0-a45a-34d7e50b7dd2\" (UID: \"17e55bd6-a19b-40e0-a45a-34d7e50b7dd2\") "
Aug 04 18:53:41.220788 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:53:41.220566    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/17e55bd6-a19b-40e0-a45a-34d7e50b7dd2-cabundle-cert\") pod \"17e55bd6-a19b-40e0-a45a-34d7e50b7dd2\" (UID: \"17e55bd6-a19b-40e0-a45a-34d7e50b7dd2\") "
Aug 04 18:53:41.220788 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:53:41.220737    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17e55bd6-a19b-40e0-a45a-34d7e50b7dd2-model-cache" (OuterVolumeSpecName: "model-cache") pod "17e55bd6-a19b-40e0-a45a-34d7e50b7dd2" (UID: "17e55bd6-a19b-40e0-a45a-34d7e50b7dd2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:53:41.220868 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:53:41.220795    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17e55bd6-a19b-40e0-a45a-34d7e50b7dd2-tmp-dir\") pod \"17e55bd6-a19b-40e0-a45a-34d7e50b7dd2\" (UID: \"17e55bd6-a19b-40e0-a45a-34d7e50b7dd2\") "
Aug 04 18:53:41.220868 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:53:41.220847    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tfwsn\" (UniqueName: \"kubernetes.io/projected/17e55bd6-a19b-40e0-a45a-34d7e50b7dd2-kube-api-access-tfwsn\") pod \"17e55bd6-a19b-40e0-a45a-34d7e50b7dd2\" (UID: \"17e55bd6-a19b-40e0-a45a-34d7e50b7dd2\") "
Aug 04 18:53:41.220966 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:53:41.220887    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17e55bd6-a19b-40e0-a45a-34d7e50b7dd2-home\") pod \"17e55bd6-a19b-40e0-a45a-34d7e50b7dd2\" (UID: \"17e55bd6-a19b-40e0-a45a-34d7e50b7dd2\") "
Aug 04 18:53:41.220966 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:53:41.220909    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17e55bd6-a19b-40e0-a45a-34d7e50b7dd2-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "17e55bd6-a19b-40e0-a45a-34d7e50b7dd2" (UID: "17e55bd6-a19b-40e0-a45a-34d7e50b7dd2"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:53:41.220966 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:53:41.220914    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17e55bd6-a19b-40e0-a45a-34d7e50b7dd2-dshm\") pod \"17e55bd6-a19b-40e0-a45a-34d7e50b7dd2\" (UID: \"17e55bd6-a19b-40e0-a45a-34d7e50b7dd2\") "
Aug 04 18:53:41.221388 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:53:41.221365    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17e55bd6-a19b-40e0-a45a-34d7e50b7dd2-model-cache\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:53:41.221601 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:53:41.221391    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/17e55bd6-a19b-40e0-a45a-34d7e50b7dd2-cabundle-cert\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:53:41.221808 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:53:41.221783    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17e55bd6-a19b-40e0-a45a-34d7e50b7dd2-home" (OuterVolumeSpecName: "home") pod "17e55bd6-a19b-40e0-a45a-34d7e50b7dd2" (UID: "17e55bd6-a19b-40e0-a45a-34d7e50b7dd2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:53:41.223144 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:53:41.223118    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17e55bd6-a19b-40e0-a45a-34d7e50b7dd2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "17e55bd6-a19b-40e0-a45a-34d7e50b7dd2" (UID: "17e55bd6-a19b-40e0-a45a-34d7e50b7dd2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:53:41.223422 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:53:41.223405    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17e55bd6-a19b-40e0-a45a-34d7e50b7dd2-dshm" (OuterVolumeSpecName: "dshm") pod "17e55bd6-a19b-40e0-a45a-34d7e50b7dd2" (UID: "17e55bd6-a19b-40e0-a45a-34d7e50b7dd2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:53:41.223508 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:53:41.223433    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17e55bd6-a19b-40e0-a45a-34d7e50b7dd2-kube-api-access-tfwsn" (OuterVolumeSpecName: "kube-api-access-tfwsn") pod "17e55bd6-a19b-40e0-a45a-34d7e50b7dd2" (UID: "17e55bd6-a19b-40e0-a45a-34d7e50b7dd2"). InnerVolumeSpecName "kube-api-access-tfwsn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:53:41.232439 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:53:41.232382    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17e55bd6-a19b-40e0-a45a-34d7e50b7dd2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "17e55bd6-a19b-40e0-a45a-34d7e50b7dd2" (UID: "17e55bd6-a19b-40e0-a45a-34d7e50b7dd2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:53:41.244255 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:53:41.244229    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17e55bd6-a19b-40e0-a45a-34d7e50b7dd2-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "17e55bd6-a19b-40e0-a45a-34d7e50b7dd2" (UID: "17e55bd6-a19b-40e0-a45a-34d7e50b7dd2"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:53:41.322092 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:53:41.322050    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17e55bd6-a19b-40e0-a45a-34d7e50b7dd2-tmp-dir\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:53:41.322092 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:53:41.322082    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tfwsn\" (UniqueName: \"kubernetes.io/projected/17e55bd6-a19b-40e0-a45a-34d7e50b7dd2-kube-api-access-tfwsn\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:53:41.322092 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:53:41.322091    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17e55bd6-a19b-40e0-a45a-34d7e50b7dd2-home\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:53:41.322092 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:53:41.322099    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17e55bd6-a19b-40e0-a45a-34d7e50b7dd2-dshm\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:53:41.322357 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:53:41.322107    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17e55bd6-a19b-40e0-a45a-34d7e50b7dd2-tls-certs\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:53:41.322357 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:53:41.322117    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/17e55bd6-a19b-40e0-a45a-34d7e50b7dd2-kserve-provision-location\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:53:41.870886 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:53:41.870853    2583 generic.go:356] "Generic (PLEG): container finished" podID="17e55bd6-a19b-40e0-a45a-34d7e50b7dd2" containerID="09b2783106af4791581c0a237a7c895b087ca2ea43af77eacb0caefbfde54363" exitCode=0
Aug 04 18:53:41.871325 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:53:41.870907    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6bffc6d99d-lshf2"
Aug 04 18:53:41.871325 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:53:41.870923    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6bffc6d99d-lshf2" event={"ID":"17e55bd6-a19b-40e0-a45a-34d7e50b7dd2","Type":"ContainerDied","Data":"09b2783106af4791581c0a237a7c895b087ca2ea43af77eacb0caefbfde54363"}
Aug 04 18:53:41.871325 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:53:41.870972    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6bffc6d99d-lshf2" event={"ID":"17e55bd6-a19b-40e0-a45a-34d7e50b7dd2","Type":"ContainerDied","Data":"83654777d04c5bafc7b5f462b89d436ffd971da75a6e46f6963f53e52c057e83"}
Aug 04 18:53:41.871325 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:53:41.870997    2583 scope.go:117] "RemoveContainer" containerID="09b2783106af4791581c0a237a7c895b087ca2ea43af77eacb0caefbfde54363"
Aug 04 18:53:41.880821 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:53:41.880804    2583 scope.go:117] "RemoveContainer" containerID="c5c5c030d0e0aa0a705db47e4d0099baf5fab9660c3bb08a40682e1ba863c715"
Aug 04 18:53:41.888565 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:53:41.888547    2583 scope.go:117] "RemoveContainer" containerID="09b2783106af4791581c0a237a7c895b087ca2ea43af77eacb0caefbfde54363"
Aug 04 18:53:41.888806 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:53:41.888786    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09b2783106af4791581c0a237a7c895b087ca2ea43af77eacb0caefbfde54363\": container with ID starting with 09b2783106af4791581c0a237a7c895b087ca2ea43af77eacb0caefbfde54363 not found: ID does not exist" containerID="09b2783106af4791581c0a237a7c895b087ca2ea43af77eacb0caefbfde54363"
Aug 04 18:53:41.888894 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:53:41.888817    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09b2783106af4791581c0a237a7c895b087ca2ea43af77eacb0caefbfde54363"} err="failed to get container status \"09b2783106af4791581c0a237a7c895b087ca2ea43af77eacb0caefbfde54363\": rpc error: code = NotFound desc = could not find container \"09b2783106af4791581c0a237a7c895b087ca2ea43af77eacb0caefbfde54363\": container with ID starting with 09b2783106af4791581c0a237a7c895b087ca2ea43af77eacb0caefbfde54363 not found: ID does not exist"
Aug 04 18:53:41.888894 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:53:41.888843    2583 scope.go:117] "RemoveContainer" containerID="c5c5c030d0e0aa0a705db47e4d0099baf5fab9660c3bb08a40682e1ba863c715"
Aug 04 18:53:41.889078 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:53:41.889060    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5c5c030d0e0aa0a705db47e4d0099baf5fab9660c3bb08a40682e1ba863c715\": container with ID starting with c5c5c030d0e0aa0a705db47e4d0099baf5fab9660c3bb08a40682e1ba863c715 not found: ID does not exist" containerID="c5c5c030d0e0aa0a705db47e4d0099baf5fab9660c3bb08a40682e1ba863c715"
Aug 04 18:53:41.889118 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:53:41.889085    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5c5c030d0e0aa0a705db47e4d0099baf5fab9660c3bb08a40682e1ba863c715"} err="failed to get container status \"c5c5c030d0e0aa0a705db47e4d0099baf5fab9660c3bb08a40682e1ba863c715\": rpc error: code = NotFound desc = could not find container \"c5c5c030d0e0aa0a705db47e4d0099baf5fab9660c3bb08a40682e1ba863c715\": container with ID starting with c5c5c030d0e0aa0a705db47e4d0099baf5fab9660c3bb08a40682e1ba863c715 not found: ID does not exist"
Aug 04 18:53:41.895860 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:53:41.895837    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6bffc6d99d-lshf2"]
Aug 04 18:53:41.899601 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:53:41.899580    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-6bffc6d99d-lshf2"]
Aug 04 18:53:42.942483 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:53:42.942433    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="17e55bd6-a19b-40e0-a45a-34d7e50b7dd2" path="/var/lib/kubelet/pods/17e55bd6-a19b-40e0-a45a-34d7e50b7dd2/volumes"
Aug 04 18:54:11.224961 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:54:11.224923    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58t94tx"]
Aug 04 18:54:11.225619 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:54:11.225363    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17e55bd6-a19b-40e0-a45a-34d7e50b7dd2" containerName="storage-initializer"
Aug 04 18:54:11.225619 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:54:11.225384    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="17e55bd6-a19b-40e0-a45a-34d7e50b7dd2" containerName="storage-initializer"
Aug 04 18:54:11.225619 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:54:11.225421    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17e55bd6-a19b-40e0-a45a-34d7e50b7dd2" containerName="main"
Aug 04 18:54:11.225619 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:54:11.225428    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="17e55bd6-a19b-40e0-a45a-34d7e50b7dd2" containerName="main"
Aug 04 18:54:11.225619 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:54:11.225510    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="17e55bd6-a19b-40e0-a45a-34d7e50b7dd2" containerName="main"
Aug 04 18:54:11.228806 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:54:11.228781    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58t94tx"
Aug 04 18:54:11.231713 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:54:11.231689    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"default-dockercfg-czcbj\""
Aug 04 18:54:11.231854 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:54:11.231719    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"openshift-service-ca.crt\""
Aug 04 18:54:11.231854 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:54:11.231739    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"seaweedfs-s3-creds\""
Aug 04 18:54:11.231965 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:54:11.231952    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"odh-kserve-custom-ca-bundle\""
Aug 04 18:54:11.232253 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:54:11.232239    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"llmisv44d181485fad85e662eb092f3749502f-kserve-self-signed-certs\""
Aug 04 18:54:11.233152 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:54:11.233136    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"kube-root-ca.crt\""
Aug 04 18:54:11.244349 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:54:11.244321    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58t94tx"]
Aug 04 18:54:11.379882 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:54:11.379843    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a14cea9b-18a7-472e-b455-284366ae3501-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58t94tx\" (UID: \"a14cea9b-18a7-472e-b455-284366ae3501\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58t94tx"
Aug 04 18:54:11.380094 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:54:11.379889    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a14cea9b-18a7-472e-b455-284366ae3501-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58t94tx\" (UID: \"a14cea9b-18a7-472e-b455-284366ae3501\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58t94tx"
Aug 04 18:54:11.380094 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:54:11.379913    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a14cea9b-18a7-472e-b455-284366ae3501-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58t94tx\" (UID: \"a14cea9b-18a7-472e-b455-284366ae3501\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58t94tx"
Aug 04 18:54:11.380094 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:54:11.379929    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a14cea9b-18a7-472e-b455-284366ae3501-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58t94tx\" (UID: \"a14cea9b-18a7-472e-b455-284366ae3501\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58t94tx"
Aug 04 18:54:11.380094 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:54:11.379999    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a14cea9b-18a7-472e-b455-284366ae3501-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58t94tx\" (UID: \"a14cea9b-18a7-472e-b455-284366ae3501\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58t94tx"
Aug 04 18:54:11.380094 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:54:11.380042    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a14cea9b-18a7-472e-b455-284366ae3501-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58t94tx\" (UID: \"a14cea9b-18a7-472e-b455-284366ae3501\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58t94tx"
Aug 04 18:54:11.380094 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:54:11.380089    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qr95m\" (UniqueName: \"kubernetes.io/projected/a14cea9b-18a7-472e-b455-284366ae3501-kube-api-access-qr95m\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58t94tx\" (UID: \"a14cea9b-18a7-472e-b455-284366ae3501\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58t94tx"
Aug 04 18:54:11.380418 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:54:11.380126    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a14cea9b-18a7-472e-b455-284366ae3501-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58t94tx\" (UID: \"a14cea9b-18a7-472e-b455-284366ae3501\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58t94tx"
Aug 04 18:54:11.481506 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:54:11.481383    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a14cea9b-18a7-472e-b455-284366ae3501-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58t94tx\" (UID: \"a14cea9b-18a7-472e-b455-284366ae3501\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58t94tx"
Aug 04 18:54:11.481506 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:54:11.481431    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a14cea9b-18a7-472e-b455-284366ae3501-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58t94tx\" (UID: \"a14cea9b-18a7-472e-b455-284366ae3501\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58t94tx"
Aug 04 18:54:11.481506 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:54:11.481459    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a14cea9b-18a7-472e-b455-284366ae3501-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58t94tx\" (UID: \"a14cea9b-18a7-472e-b455-284366ae3501\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58t94tx"
Aug 04 18:54:11.481506 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:54:11.481505    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a14cea9b-18a7-472e-b455-284366ae3501-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58t94tx\" (UID: \"a14cea9b-18a7-472e-b455-284366ae3501\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58t94tx"
Aug 04 18:54:11.481867 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:54:11.481531    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a14cea9b-18a7-472e-b455-284366ae3501-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58t94tx\" (UID: \"a14cea9b-18a7-472e-b455-284366ae3501\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58t94tx"
Aug 04 18:54:11.481867 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:54:11.481562    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a14cea9b-18a7-472e-b455-284366ae3501-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58t94tx\" (UID: \"a14cea9b-18a7-472e-b455-284366ae3501\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58t94tx"
Aug 04 18:54:11.481867 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:54:11.481596    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qr95m\" (UniqueName: \"kubernetes.io/projected/a14cea9b-18a7-472e-b455-284366ae3501-kube-api-access-qr95m\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58t94tx\" (UID: \"a14cea9b-18a7-472e-b455-284366ae3501\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58t94tx"
Aug 04 18:54:11.481867 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:54:11.481662    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a14cea9b-18a7-472e-b455-284366ae3501-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58t94tx\" (UID: \"a14cea9b-18a7-472e-b455-284366ae3501\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58t94tx"
Aug 04 18:54:11.482069 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:54:11.481922    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a14cea9b-18a7-472e-b455-284366ae3501-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58t94tx\" (UID: \"a14cea9b-18a7-472e-b455-284366ae3501\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58t94tx"
Aug 04 18:54:11.482069 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:54:11.481974    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a14cea9b-18a7-472e-b455-284366ae3501-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58t94tx\" (UID: \"a14cea9b-18a7-472e-b455-284366ae3501\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58t94tx"
Aug 04 18:54:11.482187 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:54:11.482065    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a14cea9b-18a7-472e-b455-284366ae3501-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58t94tx\" (UID: \"a14cea9b-18a7-472e-b455-284366ae3501\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58t94tx"
Aug 04 18:54:11.482187 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:54:11.482105    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a14cea9b-18a7-472e-b455-284366ae3501-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58t94tx\" (UID: \"a14cea9b-18a7-472e-b455-284366ae3501\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58t94tx"
Aug 04 18:54:11.482280 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:54:11.482259    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a14cea9b-18a7-472e-b455-284366ae3501-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58t94tx\" (UID: \"a14cea9b-18a7-472e-b455-284366ae3501\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58t94tx"
Aug 04 18:54:11.483926 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:54:11.483899    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a14cea9b-18a7-472e-b455-284366ae3501-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58t94tx\" (UID: \"a14cea9b-18a7-472e-b455-284366ae3501\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58t94tx"
Aug 04 18:54:11.484106 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:54:11.484089    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a14cea9b-18a7-472e-b455-284366ae3501-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58t94tx\" (UID: \"a14cea9b-18a7-472e-b455-284366ae3501\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58t94tx"
Aug 04 18:54:11.499607 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:54:11.499575    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qr95m\" (UniqueName: \"kubernetes.io/projected/a14cea9b-18a7-472e-b455-284366ae3501-kube-api-access-qr95m\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58t94tx\" (UID: \"a14cea9b-18a7-472e-b455-284366ae3501\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58t94tx"
Aug 04 18:54:11.539454 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:54:11.539416    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58t94tx"
Aug 04 18:54:11.687444 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:54:11.687414    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58t94tx"]
Aug 04 18:54:11.688926 ip-10-0-135-124 kubenswrapper[2583]: W0804 18:54:11.688897    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda14cea9b_18a7_472e_b455_284366ae3501.slice/crio-842bbb51d705dbf3fc3b0268beddcba811391ac977c0f509a1dbb050f8b60ad6 WatchSource:0}: Error finding container 842bbb51d705dbf3fc3b0268beddcba811391ac977c0f509a1dbb050f8b60ad6: Status 404 returned error can't find the container with id 842bbb51d705dbf3fc3b0268beddcba811391ac977c0f509a1dbb050f8b60ad6
Aug 04 18:54:12.041259 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:54:12.041160    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58t94tx" event={"ID":"a14cea9b-18a7-472e-b455-284366ae3501","Type":"ContainerStarted","Data":"273e09442cf3ece4b001ceaa4f1b7b87316f41057723db11bd1bec83cdf448d0"}
Aug 04 18:54:12.041259 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:54:12.041216    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58t94tx" event={"ID":"a14cea9b-18a7-472e-b455-284366ae3501","Type":"ContainerStarted","Data":"842bbb51d705dbf3fc3b0268beddcba811391ac977c0f509a1dbb050f8b60ad6"}
Aug 04 18:54:14.056158 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:54:14.056120    2583 generic.go:356] "Generic (PLEG): container finished" podID="a14cea9b-18a7-472e-b455-284366ae3501" containerID="273e09442cf3ece4b001ceaa4f1b7b87316f41057723db11bd1bec83cdf448d0" exitCode=0
Aug 04 18:54:14.056657 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:54:14.056212    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58t94tx" event={"ID":"a14cea9b-18a7-472e-b455-284366ae3501","Type":"ContainerDied","Data":"273e09442cf3ece4b001ceaa4f1b7b87316f41057723db11bd1bec83cdf448d0"}
Aug 04 18:54:15.064838 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:54:15.064800    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58t94tx" event={"ID":"a14cea9b-18a7-472e-b455-284366ae3501","Type":"ContainerStarted","Data":"a3a410603c34988606921ebd96ddd1b77cc3cfe29edbde11a72cfe1a67e2a7ab"}
Aug 04 18:54:15.092239 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:54:15.092186    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58t94tx" podStartSLOduration=1.8960054240000002 podStartE2EDuration="4.092169084s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.19616366s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:54:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:54:15.091322651 +0000 UTC m=+3268.672541228" watchObservedRunningTime="2026-08-04 18:54:15.092169084 +0000 UTC m=+3268.673387661"
Aug 04 18:54:21.540084 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:54:21.540039    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58t94tx"
Aug 04 18:54:21.540739 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:54:21.540116    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58t94tx"
Aug 04 18:54:21.541534 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:54:21.541501    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58t94tx" podUID="a14cea9b-18a7-472e-b455-284366ae3501" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/health\": dial tcp 10.132.0.83:8000: connect: connection refused"
Aug 04 18:54:31.539986 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:54:31.539945    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58t94tx" podUID="a14cea9b-18a7-472e-b455-284366ae3501" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/health\": dial tcp 10.132.0.83:8000: connect: connection refused"
Aug 04 18:54:41.540085 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:54:41.540037    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58t94tx" podUID="a14cea9b-18a7-472e-b455-284366ae3501" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/health\": dial tcp 10.132.0.83:8000: connect: connection refused"
Aug 04 18:54:47.224293 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:54:47.224258    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-wskfb_6157c2d3-2fd0-4544-88ad-434ac75f8c04/console-operator/1.log"
Aug 04 18:54:47.237977 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:54:47.237948    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-wskfb_6157c2d3-2fd0-4544-88ad-434ac75f8c04/console-operator/1.log"
Aug 04 18:54:51.540072 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:54:51.540019    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58t94tx" podUID="a14cea9b-18a7-472e-b455-284366ae3501" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/health\": dial tcp 10.132.0.83:8000: connect: connection refused"
Aug 04 18:55:01.540535 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:55:01.540448    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58t94tx" podUID="a14cea9b-18a7-472e-b455-284366ae3501" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/health\": dial tcp 10.132.0.83:8000: connect: connection refused"
Aug 04 18:55:11.540952 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:55:11.540849    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58t94tx" podUID="a14cea9b-18a7-472e-b455-284366ae3501" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/health\": dial tcp 10.132.0.83:8000: connect: connection refused"
Aug 04 18:55:21.539873 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:55:21.539820    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58t94tx" podUID="a14cea9b-18a7-472e-b455-284366ae3501" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/health\": dial tcp 10.132.0.83:8000: connect: connection refused"
Aug 04 18:55:31.540939 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:55:31.540881    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58t94tx" podUID="a14cea9b-18a7-472e-b455-284366ae3501" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/health\": dial tcp 10.132.0.83:8000: connect: connection refused"
Aug 04 18:55:41.540179 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:55:41.540130    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58t94tx" podUID="a14cea9b-18a7-472e-b455-284366ae3501" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/health\": dial tcp 10.132.0.83:8000: connect: connection refused"
Aug 04 18:55:51.540323 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:55:51.540272    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58t94tx" podUID="a14cea9b-18a7-472e-b455-284366ae3501" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/health\": dial tcp 10.132.0.83:8000: connect: connection refused"
Aug 04 18:56:01.540246 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:56:01.540200    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58t94tx" podUID="a14cea9b-18a7-472e-b455-284366ae3501" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/health\": dial tcp 10.132.0.83:8000: connect: connection refused"
Aug 04 18:56:11.551183 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:56:11.551146    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58t94tx"
Aug 04 18:56:11.559053 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:56:11.559020    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58t94tx"
Aug 04 18:56:50.450316 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:56:50.450276    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58t94tx"]
Aug 04 18:57:05.473180 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:57:05.473133    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58t94tx" podUID="a14cea9b-18a7-472e-b455-284366ae3501" containerName="main" containerID="cri-o://a3a410603c34988606921ebd96ddd1b77cc3cfe29edbde11a72cfe1a67e2a7ab" gracePeriod=45
Aug 04 18:57:06.216020 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:57:06.215991    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58t94tx"
Aug 04 18:57:06.237465 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:57:06.237436    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a14cea9b-18a7-472e-b455-284366ae3501-tmp-dir\") pod \"a14cea9b-18a7-472e-b455-284366ae3501\" (UID: \"a14cea9b-18a7-472e-b455-284366ae3501\") "
Aug 04 18:57:06.237628 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:57:06.237490    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a14cea9b-18a7-472e-b455-284366ae3501-cabundle-cert\") pod \"a14cea9b-18a7-472e-b455-284366ae3501\" (UID: \"a14cea9b-18a7-472e-b455-284366ae3501\") "
Aug 04 18:57:06.237628 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:57:06.237554    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a14cea9b-18a7-472e-b455-284366ae3501-kserve-provision-location\") pod \"a14cea9b-18a7-472e-b455-284366ae3501\" (UID: \"a14cea9b-18a7-472e-b455-284366ae3501\") "
Aug 04 18:57:06.237752 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:57:06.237647    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a14cea9b-18a7-472e-b455-284366ae3501-dshm\") pod \"a14cea9b-18a7-472e-b455-284366ae3501\" (UID: \"a14cea9b-18a7-472e-b455-284366ae3501\") "
Aug 04 18:57:06.237752 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:57:06.237687    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a14cea9b-18a7-472e-b455-284366ae3501-tls-certs\") pod \"a14cea9b-18a7-472e-b455-284366ae3501\" (UID: \"a14cea9b-18a7-472e-b455-284366ae3501\") "
Aug 04 18:57:06.237752 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:57:06.237743    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a14cea9b-18a7-472e-b455-284366ae3501-model-cache\") pod \"a14cea9b-18a7-472e-b455-284366ae3501\" (UID: \"a14cea9b-18a7-472e-b455-284366ae3501\") "
Aug 04 18:57:06.237911 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:57:06.237814    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qr95m\" (UniqueName: \"kubernetes.io/projected/a14cea9b-18a7-472e-b455-284366ae3501-kube-api-access-qr95m\") pod \"a14cea9b-18a7-472e-b455-284366ae3501\" (UID: \"a14cea9b-18a7-472e-b455-284366ae3501\") "
Aug 04 18:57:06.237911 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:57:06.237849    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a14cea9b-18a7-472e-b455-284366ae3501-home\") pod \"a14cea9b-18a7-472e-b455-284366ae3501\" (UID: \"a14cea9b-18a7-472e-b455-284366ae3501\") "
Aug 04 18:57:06.238278 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:57:06.238107    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a14cea9b-18a7-472e-b455-284366ae3501-model-cache" (OuterVolumeSpecName: "model-cache") pod "a14cea9b-18a7-472e-b455-284366ae3501" (UID: "a14cea9b-18a7-472e-b455-284366ae3501"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:57:06.238278 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:57:06.238220    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a14cea9b-18a7-472e-b455-284366ae3501-model-cache\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:57:06.238842 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:57:06.238569    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a14cea9b-18a7-472e-b455-284366ae3501-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a14cea9b-18a7-472e-b455-284366ae3501" (UID: "a14cea9b-18a7-472e-b455-284366ae3501"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:57:06.238842 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:57:06.238795    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a14cea9b-18a7-472e-b455-284366ae3501-home" (OuterVolumeSpecName: "home") pod "a14cea9b-18a7-472e-b455-284366ae3501" (UID: "a14cea9b-18a7-472e-b455-284366ae3501"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:57:06.241566 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:57:06.241525    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a14cea9b-18a7-472e-b455-284366ae3501-dshm" (OuterVolumeSpecName: "dshm") pod "a14cea9b-18a7-472e-b455-284366ae3501" (UID: "a14cea9b-18a7-472e-b455-284366ae3501"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:57:06.241786 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:57:06.241761    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a14cea9b-18a7-472e-b455-284366ae3501-kube-api-access-qr95m" (OuterVolumeSpecName: "kube-api-access-qr95m") pod "a14cea9b-18a7-472e-b455-284366ae3501" (UID: "a14cea9b-18a7-472e-b455-284366ae3501"). InnerVolumeSpecName "kube-api-access-qr95m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:57:06.242401 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:57:06.242367    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a14cea9b-18a7-472e-b455-284366ae3501-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a14cea9b-18a7-472e-b455-284366ae3501" (UID: "a14cea9b-18a7-472e-b455-284366ae3501"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:57:06.252174 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:57:06.252145    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a14cea9b-18a7-472e-b455-284366ae3501-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a14cea9b-18a7-472e-b455-284366ae3501" (UID: "a14cea9b-18a7-472e-b455-284366ae3501"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:57:06.267288 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:57:06.267225    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a14cea9b-18a7-472e-b455-284366ae3501-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a14cea9b-18a7-472e-b455-284366ae3501" (UID: "a14cea9b-18a7-472e-b455-284366ae3501"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:57:06.339610 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:57:06.339571    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a14cea9b-18a7-472e-b455-284366ae3501-dshm\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:57:06.339610 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:57:06.339600    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a14cea9b-18a7-472e-b455-284366ae3501-tls-certs\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:57:06.339610 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:57:06.339610    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qr95m\" (UniqueName: \"kubernetes.io/projected/a14cea9b-18a7-472e-b455-284366ae3501-kube-api-access-qr95m\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:57:06.339610 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:57:06.339619    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a14cea9b-18a7-472e-b455-284366ae3501-home\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:57:06.339896 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:57:06.339628    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a14cea9b-18a7-472e-b455-284366ae3501-tmp-dir\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:57:06.339896 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:57:06.339638    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a14cea9b-18a7-472e-b455-284366ae3501-cabundle-cert\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:57:06.339896 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:57:06.339645    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a14cea9b-18a7-472e-b455-284366ae3501-kserve-provision-location\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 18:57:07.004274 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:57:07.004237    2583 generic.go:356] "Generic (PLEG): container finished" podID="a14cea9b-18a7-472e-b455-284366ae3501" containerID="a3a410603c34988606921ebd96ddd1b77cc3cfe29edbde11a72cfe1a67e2a7ab" exitCode=0
Aug 04 18:57:07.004693 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:57:07.004285    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58t94tx"
Aug 04 18:57:07.004693 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:57:07.004321    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58t94tx" event={"ID":"a14cea9b-18a7-472e-b455-284366ae3501","Type":"ContainerDied","Data":"a3a410603c34988606921ebd96ddd1b77cc3cfe29edbde11a72cfe1a67e2a7ab"}
Aug 04 18:57:07.004693 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:57:07.004377    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58t94tx" event={"ID":"a14cea9b-18a7-472e-b455-284366ae3501","Type":"ContainerDied","Data":"842bbb51d705dbf3fc3b0268beddcba811391ac977c0f509a1dbb050f8b60ad6"}
Aug 04 18:57:07.004693 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:57:07.004400    2583 scope.go:117] "RemoveContainer" containerID="a3a410603c34988606921ebd96ddd1b77cc3cfe29edbde11a72cfe1a67e2a7ab"
Aug 04 18:57:07.013686 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:57:07.013662    2583 scope.go:117] "RemoveContainer" containerID="273e09442cf3ece4b001ceaa4f1b7b87316f41057723db11bd1bec83cdf448d0"
Aug 04 18:57:07.021615 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:57:07.021581    2583 scope.go:117] "RemoveContainer" containerID="a3a410603c34988606921ebd96ddd1b77cc3cfe29edbde11a72cfe1a67e2a7ab"
Aug 04 18:57:07.021878 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:57:07.021858    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3a410603c34988606921ebd96ddd1b77cc3cfe29edbde11a72cfe1a67e2a7ab\": container with ID starting with a3a410603c34988606921ebd96ddd1b77cc3cfe29edbde11a72cfe1a67e2a7ab not found: ID does not exist" containerID="a3a410603c34988606921ebd96ddd1b77cc3cfe29edbde11a72cfe1a67e2a7ab"
Aug 04 18:57:07.021931 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:57:07.021887    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3a410603c34988606921ebd96ddd1b77cc3cfe29edbde11a72cfe1a67e2a7ab"} err="failed to get container status \"a3a410603c34988606921ebd96ddd1b77cc3cfe29edbde11a72cfe1a67e2a7ab\": rpc error: code = NotFound desc = could not find container \"a3a410603c34988606921ebd96ddd1b77cc3cfe29edbde11a72cfe1a67e2a7ab\": container with ID starting with a3a410603c34988606921ebd96ddd1b77cc3cfe29edbde11a72cfe1a67e2a7ab not found: ID does not exist"
Aug 04 18:57:07.021931 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:57:07.021906    2583 scope.go:117] "RemoveContainer" containerID="273e09442cf3ece4b001ceaa4f1b7b87316f41057723db11bd1bec83cdf448d0"
Aug 04 18:57:07.022130 ip-10-0-135-124 kubenswrapper[2583]: E0804 18:57:07.022112    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"273e09442cf3ece4b001ceaa4f1b7b87316f41057723db11bd1bec83cdf448d0\": container with ID starting with 273e09442cf3ece4b001ceaa4f1b7b87316f41057723db11bd1bec83cdf448d0 not found: ID does not exist" containerID="273e09442cf3ece4b001ceaa4f1b7b87316f41057723db11bd1bec83cdf448d0"
Aug 04 18:57:07.022174 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:57:07.022136    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"273e09442cf3ece4b001ceaa4f1b7b87316f41057723db11bd1bec83cdf448d0"} err="failed to get container status \"273e09442cf3ece4b001ceaa4f1b7b87316f41057723db11bd1bec83cdf448d0\": rpc error: code = NotFound desc = could not find container \"273e09442cf3ece4b001ceaa4f1b7b87316f41057723db11bd1bec83cdf448d0\": container with ID starting with 273e09442cf3ece4b001ceaa4f1b7b87316f41057723db11bd1bec83cdf448d0 not found: ID does not exist"
Aug 04 18:57:07.033014 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:57:07.032976    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58t94tx"]
Aug 04 18:57:07.033996 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:57:07.033973    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-58t94tx"]
Aug 04 18:57:08.942519 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:57:08.942483    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a14cea9b-18a7-472e-b455-284366ae3501" path="/var/lib/kubelet/pods/a14cea9b-18a7-472e-b455-284366ae3501/volumes"
Aug 04 18:57:19.525566 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:57:19.525527    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-759d7d7b7c-jw68j"]
Aug 04 18:57:19.526042 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:57:19.526030    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a14cea9b-18a7-472e-b455-284366ae3501" containerName="main"
Aug 04 18:57:19.526115 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:57:19.526050    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="a14cea9b-18a7-472e-b455-284366ae3501" containerName="main"
Aug 04 18:57:19.526115 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:57:19.526065    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a14cea9b-18a7-472e-b455-284366ae3501" containerName="storage-initializer"
Aug 04 18:57:19.526115 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:57:19.526076    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="a14cea9b-18a7-472e-b455-284366ae3501" containerName="storage-initializer"
Aug 04 18:57:19.526266 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:57:19.526187    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="a14cea9b-18a7-472e-b455-284366ae3501" containerName="main"
Aug 04 18:57:19.529422 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:57:19.529399    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-759d7d7b7c-jw68j"
Aug 04 18:57:19.533659 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:57:19.533635    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"seaweedfs-s3-creds\""
Aug 04 18:57:19.533779 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:57:19.533682    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"openshift-service-ca.crt\""
Aug 04 18:57:19.536573 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:57:19.536223    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"odh-kserve-custom-ca-bundle\""
Aug 04 18:57:19.536573 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:57:19.536414    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"kube-root-ca.crt\""
Aug 04 18:57:19.536573 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:57:19.536445    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"custom-route-timeout-pd-test-kserve-self-signed-certs\""
Aug 04 18:57:19.536725 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:57:19.536610    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"default-dockercfg-vglqf\""
Aug 04 18:57:19.554875 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:57:19.554846    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-759d7d7b7c-jw68j"]
Aug 04 18:57:19.656085 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:57:19.656047    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b849d68f-a68e-4a78-809b-540472fbc43b-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-prefill-759d7d7b7c-jw68j\" (UID: \"b849d68f-a68e-4a78-809b-540472fbc43b\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-759d7d7b7c-jw68j"
Aug 04 18:57:19.656290 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:57:19.656099    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b849d68f-a68e-4a78-809b-540472fbc43b-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-prefill-759d7d7b7c-jw68j\" (UID: \"b849d68f-a68e-4a78-809b-540472fbc43b\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-759d7d7b7c-jw68j"
Aug 04 18:57:19.656290 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:57:19.656157    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b849d68f-a68e-4a78-809b-540472fbc43b-dshm\") pod \"custom-route-timeout-pd-test-kserve-prefill-759d7d7b7c-jw68j\" (UID: \"b849d68f-a68e-4a78-809b-540472fbc43b\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-759d7d7b7c-jw68j"
Aug 04 18:57:19.656290 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:57:19.656180    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b849d68f-a68e-4a78-809b-540472fbc43b-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-prefill-759d7d7b7c-jw68j\" (UID: \"b849d68f-a68e-4a78-809b-540472fbc43b\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-759d7d7b7c-jw68j"
Aug 04 18:57:19.656290 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:57:19.656256    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b849d68f-a68e-4a78-809b-540472fbc43b-home\") pod \"custom-route-timeout-pd-test-kserve-prefill-759d7d7b7c-jw68j\" (UID: \"b849d68f-a68e-4a78-809b-540472fbc43b\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-759d7d7b7c-jw68j"
Aug 04 18:57:19.656511 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:57:19.656329    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b849d68f-a68e-4a78-809b-540472fbc43b-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-prefill-759d7d7b7c-jw68j\" (UID: \"b849d68f-a68e-4a78-809b-540472fbc43b\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-759d7d7b7c-jw68j"
Aug 04 18:57:19.656511 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:57:19.656357    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b849d68f-a68e-4a78-809b-540472fbc43b-model-cache\") pod \"custom-route-timeout-pd-test-kserve-prefill-759d7d7b7c-jw68j\" (UID: \"b849d68f-a68e-4a78-809b-540472fbc43b\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-759d7d7b7c-jw68j"
Aug 04 18:57:19.656511 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:57:19.656378    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gv8nh\" (UniqueName: \"kubernetes.io/projected/b849d68f-a68e-4a78-809b-540472fbc43b-kube-api-access-gv8nh\") pod \"custom-route-timeout-pd-test-kserve-prefill-759d7d7b7c-jw68j\" (UID: \"b849d68f-a68e-4a78-809b-540472fbc43b\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-759d7d7b7c-jw68j"
Aug 04 18:57:19.757159 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:57:19.757118    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b849d68f-a68e-4a78-809b-540472fbc43b-home\") pod \"custom-route-timeout-pd-test-kserve-prefill-759d7d7b7c-jw68j\" (UID: \"b849d68f-a68e-4a78-809b-540472fbc43b\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-759d7d7b7c-jw68j"
Aug 04 18:57:19.757370 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:57:19.757185    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b849d68f-a68e-4a78-809b-540472fbc43b-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-prefill-759d7d7b7c-jw68j\" (UID: \"b849d68f-a68e-4a78-809b-540472fbc43b\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-759d7d7b7c-jw68j"
Aug 04 18:57:19.757370 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:57:19.757212    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b849d68f-a68e-4a78-809b-540472fbc43b-model-cache\") pod \"custom-route-timeout-pd-test-kserve-prefill-759d7d7b7c-jw68j\" (UID: \"b849d68f-a68e-4a78-809b-540472fbc43b\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-759d7d7b7c-jw68j"
Aug 04 18:57:19.757370 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:57:19.757233    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gv8nh\" (UniqueName: \"kubernetes.io/projected/b849d68f-a68e-4a78-809b-540472fbc43b-kube-api-access-gv8nh\") pod \"custom-route-timeout-pd-test-kserve-prefill-759d7d7b7c-jw68j\" (UID: \"b849d68f-a68e-4a78-809b-540472fbc43b\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-759d7d7b7c-jw68j"
Aug 04 18:57:19.757370 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:57:19.757279    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b849d68f-a68e-4a78-809b-540472fbc43b-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-prefill-759d7d7b7c-jw68j\" (UID: \"b849d68f-a68e-4a78-809b-540472fbc43b\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-759d7d7b7c-jw68j"
Aug 04 18:57:19.757370 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:57:19.757330    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b849d68f-a68e-4a78-809b-540472fbc43b-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-prefill-759d7d7b7c-jw68j\" (UID: \"b849d68f-a68e-4a78-809b-540472fbc43b\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-759d7d7b7c-jw68j"
Aug 04 18:57:19.757370 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:57:19.757364    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b849d68f-a68e-4a78-809b-540472fbc43b-dshm\") pod \"custom-route-timeout-pd-test-kserve-prefill-759d7d7b7c-jw68j\" (UID: \"b849d68f-a68e-4a78-809b-540472fbc43b\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-759d7d7b7c-jw68j"
Aug 04 18:57:19.757754 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:57:19.757390    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b849d68f-a68e-4a78-809b-540472fbc43b-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-prefill-759d7d7b7c-jw68j\" (UID: \"b849d68f-a68e-4a78-809b-540472fbc43b\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-759d7d7b7c-jw68j"
Aug 04 18:57:19.757754 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:57:19.757664    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b849d68f-a68e-4a78-809b-540472fbc43b-home\") pod \"custom-route-timeout-pd-test-kserve-prefill-759d7d7b7c-jw68j\" (UID: \"b849d68f-a68e-4a78-809b-540472fbc43b\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-759d7d7b7c-jw68j"
Aug 04 18:57:19.757754 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:57:19.757693    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b849d68f-a68e-4a78-809b-540472fbc43b-model-cache\") pod \"custom-route-timeout-pd-test-kserve-prefill-759d7d7b7c-jw68j\" (UID: \"b849d68f-a68e-4a78-809b-540472fbc43b\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-759d7d7b7c-jw68j"
Aug 04 18:57:19.757914 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:57:19.757749    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b849d68f-a68e-4a78-809b-540472fbc43b-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-prefill-759d7d7b7c-jw68j\" (UID: \"b849d68f-a68e-4a78-809b-540472fbc43b\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-759d7d7b7c-jw68j"
Aug 04 18:57:19.757914 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:57:19.757802    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b849d68f-a68e-4a78-809b-540472fbc43b-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-prefill-759d7d7b7c-jw68j\" (UID: \"b849d68f-a68e-4a78-809b-540472fbc43b\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-759d7d7b7c-jw68j"
Aug 04 18:57:19.758179 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:57:19.758156    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b849d68f-a68e-4a78-809b-540472fbc43b-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-prefill-759d7d7b7c-jw68j\" (UID: \"b849d68f-a68e-4a78-809b-540472fbc43b\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-759d7d7b7c-jw68j"
Aug 04 18:57:19.759813 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:57:19.759790    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b849d68f-a68e-4a78-809b-540472fbc43b-dshm\") pod \"custom-route-timeout-pd-test-kserve-prefill-759d7d7b7c-jw68j\" (UID: \"b849d68f-a68e-4a78-809b-540472fbc43b\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-759d7d7b7c-jw68j"
Aug 04 18:57:19.759890 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:57:19.759873    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b849d68f-a68e-4a78-809b-540472fbc43b-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-prefill-759d7d7b7c-jw68j\" (UID: \"b849d68f-a68e-4a78-809b-540472fbc43b\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-759d7d7b7c-jw68j"
Aug 04 18:57:19.767779 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:57:19.767757    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gv8nh\" (UniqueName: \"kubernetes.io/projected/b849d68f-a68e-4a78-809b-540472fbc43b-kube-api-access-gv8nh\") pod \"custom-route-timeout-pd-test-kserve-prefill-759d7d7b7c-jw68j\" (UID: \"b849d68f-a68e-4a78-809b-540472fbc43b\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-759d7d7b7c-jw68j"
Aug 04 18:57:19.840316 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:57:19.840221    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-759d7d7b7c-jw68j"
Aug 04 18:57:20.017259 ip-10-0-135-124 kubenswrapper[2583]: W0804 18:57:20.017209    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb849d68f_a68e_4a78_809b_540472fbc43b.slice/crio-84c585bd0c1f89f08f8a7157eeb0c551f4b9c4f79b9ba4042b0b1714ac3913e4 WatchSource:0}: Error finding container 84c585bd0c1f89f08f8a7157eeb0c551f4b9c4f79b9ba4042b0b1714ac3913e4: Status 404 returned error can't find the container with id 84c585bd0c1f89f08f8a7157eeb0c551f4b9c4f79b9ba4042b0b1714ac3913e4
Aug 04 18:57:20.018111 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:57:20.018082    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-759d7d7b7c-jw68j"]
Aug 04 18:57:20.075911 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:57:20.075882    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-759d7d7b7c-jw68j" event={"ID":"b849d68f-a68e-4a78-809b-540472fbc43b","Type":"ContainerStarted","Data":"84c585bd0c1f89f08f8a7157eeb0c551f4b9c4f79b9ba4042b0b1714ac3913e4"}
Aug 04 18:57:21.082673 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:57:21.082633    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-759d7d7b7c-jw68j" event={"ID":"b849d68f-a68e-4a78-809b-540472fbc43b","Type":"ContainerStarted","Data":"2c8324478a379b7e4ffdffd3b6247229ca7ea297c57a579f96a49f8fa4d36130"}
Aug 04 18:57:23.093535 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:57:23.093497    2583 generic.go:356] "Generic (PLEG): container finished" podID="b849d68f-a68e-4a78-809b-540472fbc43b" containerID="2c8324478a379b7e4ffdffd3b6247229ca7ea297c57a579f96a49f8fa4d36130" exitCode=0
Aug 04 18:57:23.093998 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:57:23.093559    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-759d7d7b7c-jw68j" event={"ID":"b849d68f-a68e-4a78-809b-540472fbc43b","Type":"ContainerDied","Data":"2c8324478a379b7e4ffdffd3b6247229ca7ea297c57a579f96a49f8fa4d36130"}
Aug 04 18:57:24.100866 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:57:24.100832    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-759d7d7b7c-jw68j" event={"ID":"b849d68f-a68e-4a78-809b-540472fbc43b","Type":"ContainerStarted","Data":"733c7d538f3cbfae3bcb85a2c30d8095706da61874a1a610d0c58904645204a8"}
Aug 04 18:57:24.129290 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:57:24.129237    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-759d7d7b7c-jw68j" podStartSLOduration=2.995354807 podStartE2EDuration="5.129220926s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.133866119s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:57:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:57:24.125778048 +0000 UTC m=+3457.706996625" watchObservedRunningTime="2026-08-04 18:57:24.129220926 +0000 UTC m=+3457.710439580"
Aug 04 18:57:29.841223 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:57:29.841179    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-759d7d7b7c-jw68j"
Aug 04 18:57:29.841739 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:57:29.841246    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-759d7d7b7c-jw68j"
Aug 04 18:57:29.842754 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:57:29.842726    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-759d7d7b7c-jw68j" podUID="b849d68f-a68e-4a78-809b-540472fbc43b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/health\": dial tcp 10.132.0.84:8000: connect: connection refused"
Aug 04 18:57:39.841120 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:57:39.841072    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-759d7d7b7c-jw68j" podUID="b849d68f-a68e-4a78-809b-540472fbc43b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/health\": dial tcp 10.132.0.84:8000: connect: connection refused"
Aug 04 18:57:49.841082 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:57:49.841029    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-759d7d7b7c-jw68j" podUID="b849d68f-a68e-4a78-809b-540472fbc43b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/health\": dial tcp 10.132.0.84:8000: connect: connection refused"
Aug 04 18:57:59.841632 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:57:59.841586    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-759d7d7b7c-jw68j" podUID="b849d68f-a68e-4a78-809b-540472fbc43b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/health\": dial tcp 10.132.0.84:8000: connect: connection refused"
Aug 04 18:58:09.841480 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:58:09.841384    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-759d7d7b7c-jw68j" podUID="b849d68f-a68e-4a78-809b-540472fbc43b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/health\": dial tcp 10.132.0.84:8000: connect: connection refused"
Aug 04 18:58:19.840971 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:58:19.840924    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-759d7d7b7c-jw68j" podUID="b849d68f-a68e-4a78-809b-540472fbc43b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/health\": dial tcp 10.132.0.84:8000: connect: connection refused"
Aug 04 18:58:29.841017 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:58:29.840975    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-759d7d7b7c-jw68j" podUID="b849d68f-a68e-4a78-809b-540472fbc43b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/health\": dial tcp 10.132.0.84:8000: connect: connection refused"
Aug 04 18:58:39.841235 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:58:39.841184    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-759d7d7b7c-jw68j" podUID="b849d68f-a68e-4a78-809b-540472fbc43b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/health\": dial tcp 10.132.0.84:8000: connect: connection refused"
Aug 04 18:58:49.841575 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:58:49.841531    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-759d7d7b7c-jw68j" podUID="b849d68f-a68e-4a78-809b-540472fbc43b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/health\": dial tcp 10.132.0.84:8000: connect: connection refused"
Aug 04 18:58:59.841230 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:58:59.841182    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-759d7d7b7c-jw68j" podUID="b849d68f-a68e-4a78-809b-540472fbc43b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/health\": dial tcp 10.132.0.84:8000: connect: connection refused"
Aug 04 18:59:09.841510 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:59:09.841444    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-759d7d7b7c-jw68j" podUID="b849d68f-a68e-4a78-809b-540472fbc43b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/health\": dial tcp 10.132.0.84:8000: connect: connection refused"
Aug 04 18:59:19.850986 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:59:19.850950    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-759d7d7b7c-jw68j"
Aug 04 18:59:19.858783 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:59:19.858750    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-759d7d7b7c-jw68j"
Aug 04 18:59:47.267455 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:59:47.267347    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-wskfb_6157c2d3-2fd0-4544-88ad-434ac75f8c04/console-operator/1.log"
Aug 04 18:59:47.278838 ip-10-0-135-124 kubenswrapper[2583]: I0804 18:59:47.278817    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-wskfb_6157c2d3-2fd0-4544-88ad-434ac75f8c04/console-operator/1.log"
Aug 04 19:00:08.090906 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:00:08.090872    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-765cd5b55b-tsq9t"]
Aug 04 19:00:08.094584 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:00:08.094564    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-765cd5b55b-tsq9t"
Aug 04 19:00:08.099195 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:00:08.099172    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"openshift-service-ca.crt\""
Aug 04 19:00:08.099309 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:00:08.099176    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"odh-kserve-custom-ca-bundle\""
Aug 04 19:00:08.099309 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:00:08.099265    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"seaweedfs-s3-creds\""
Aug 04 19:00:08.099309 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:00:08.099294    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"default-dockercfg-njgj7\""
Aug 04 19:00:08.099449 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:00:08.099395    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"prestop-hook-test-kserve-self-signed-certs\""
Aug 04 19:00:08.099534 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:00:08.099520    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"kube-root-ca.crt\""
Aug 04 19:00:08.110558 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:00:08.110535    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-765cd5b55b-tsq9t"]
Aug 04 19:00:08.208985 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:00:08.208949    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca52852e-7dee-46f0-a413-6f14edf620cd-tmp-dir\") pod \"prestop-hook-test-kserve-765cd5b55b-tsq9t\" (UID: \"ca52852e-7dee-46f0-a413-6f14edf620cd\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-765cd5b55b-tsq9t"
Aug 04 19:00:08.209154 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:00:08.208989    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ca52852e-7dee-46f0-a413-6f14edf620cd-kserve-provision-location\") pod \"prestop-hook-test-kserve-765cd5b55b-tsq9t\" (UID: \"ca52852e-7dee-46f0-a413-6f14edf620cd\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-765cd5b55b-tsq9t"
Aug 04 19:00:08.209154 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:00:08.209047    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca52852e-7dee-46f0-a413-6f14edf620cd-tls-certs\") pod \"prestop-hook-test-kserve-765cd5b55b-tsq9t\" (UID: \"ca52852e-7dee-46f0-a413-6f14edf620cd\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-765cd5b55b-tsq9t"
Aug 04 19:00:08.209154 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:00:08.209107    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca52852e-7dee-46f0-a413-6f14edf620cd-dshm\") pod \"prestop-hook-test-kserve-765cd5b55b-tsq9t\" (UID: \"ca52852e-7dee-46f0-a413-6f14edf620cd\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-765cd5b55b-tsq9t"
Aug 04 19:00:08.209154 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:00:08.209151    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca52852e-7dee-46f0-a413-6f14edf620cd-model-cache\") pod \"prestop-hook-test-kserve-765cd5b55b-tsq9t\" (UID: \"ca52852e-7dee-46f0-a413-6f14edf620cd\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-765cd5b55b-tsq9t"
Aug 04 19:00:08.209320 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:00:08.209168    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca52852e-7dee-46f0-a413-6f14edf620cd-home\") pod \"prestop-hook-test-kserve-765cd5b55b-tsq9t\" (UID: \"ca52852e-7dee-46f0-a413-6f14edf620cd\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-765cd5b55b-tsq9t"
Aug 04 19:00:08.209320 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:00:08.209189    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ca52852e-7dee-46f0-a413-6f14edf620cd-cabundle-cert\") pod \"prestop-hook-test-kserve-765cd5b55b-tsq9t\" (UID: \"ca52852e-7dee-46f0-a413-6f14edf620cd\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-765cd5b55b-tsq9t"
Aug 04 19:00:08.209320 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:00:08.209217    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjxxf\" (UniqueName: \"kubernetes.io/projected/ca52852e-7dee-46f0-a413-6f14edf620cd-kube-api-access-tjxxf\") pod \"prestop-hook-test-kserve-765cd5b55b-tsq9t\" (UID: \"ca52852e-7dee-46f0-a413-6f14edf620cd\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-765cd5b55b-tsq9t"
Aug 04 19:00:08.310192 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:00:08.310157    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca52852e-7dee-46f0-a413-6f14edf620cd-model-cache\") pod \"prestop-hook-test-kserve-765cd5b55b-tsq9t\" (UID: \"ca52852e-7dee-46f0-a413-6f14edf620cd\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-765cd5b55b-tsq9t"
Aug 04 19:00:08.310192 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:00:08.310190    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca52852e-7dee-46f0-a413-6f14edf620cd-home\") pod \"prestop-hook-test-kserve-765cd5b55b-tsq9t\" (UID: \"ca52852e-7dee-46f0-a413-6f14edf620cd\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-765cd5b55b-tsq9t"
Aug 04 19:00:08.310425 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:00:08.310209    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ca52852e-7dee-46f0-a413-6f14edf620cd-cabundle-cert\") pod \"prestop-hook-test-kserve-765cd5b55b-tsq9t\" (UID: \"ca52852e-7dee-46f0-a413-6f14edf620cd\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-765cd5b55b-tsq9t"
Aug 04 19:00:08.310425 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:00:08.310254    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjxxf\" (UniqueName: \"kubernetes.io/projected/ca52852e-7dee-46f0-a413-6f14edf620cd-kube-api-access-tjxxf\") pod \"prestop-hook-test-kserve-765cd5b55b-tsq9t\" (UID: \"ca52852e-7dee-46f0-a413-6f14edf620cd\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-765cd5b55b-tsq9t"
Aug 04 19:00:08.310425 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:00:08.310302    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca52852e-7dee-46f0-a413-6f14edf620cd-tmp-dir\") pod \"prestop-hook-test-kserve-765cd5b55b-tsq9t\" (UID: \"ca52852e-7dee-46f0-a413-6f14edf620cd\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-765cd5b55b-tsq9t"
Aug 04 19:00:08.310425 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:00:08.310333    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ca52852e-7dee-46f0-a413-6f14edf620cd-kserve-provision-location\") pod \"prestop-hook-test-kserve-765cd5b55b-tsq9t\" (UID: \"ca52852e-7dee-46f0-a413-6f14edf620cd\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-765cd5b55b-tsq9t"
Aug 04 19:00:08.310425 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:00:08.310360    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca52852e-7dee-46f0-a413-6f14edf620cd-tls-certs\") pod \"prestop-hook-test-kserve-765cd5b55b-tsq9t\" (UID: \"ca52852e-7dee-46f0-a413-6f14edf620cd\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-765cd5b55b-tsq9t"
Aug 04 19:00:08.310425 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:00:08.310414    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca52852e-7dee-46f0-a413-6f14edf620cd-dshm\") pod \"prestop-hook-test-kserve-765cd5b55b-tsq9t\" (UID: \"ca52852e-7dee-46f0-a413-6f14edf620cd\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-765cd5b55b-tsq9t"
Aug 04 19:00:08.310783 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:00:08.310717    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca52852e-7dee-46f0-a413-6f14edf620cd-tmp-dir\") pod \"prestop-hook-test-kserve-765cd5b55b-tsq9t\" (UID: \"ca52852e-7dee-46f0-a413-6f14edf620cd\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-765cd5b55b-tsq9t"
Aug 04 19:00:08.310783 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:00:08.310775    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ca52852e-7dee-46f0-a413-6f14edf620cd-kserve-provision-location\") pod \"prestop-hook-test-kserve-765cd5b55b-tsq9t\" (UID: \"ca52852e-7dee-46f0-a413-6f14edf620cd\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-765cd5b55b-tsq9t"
Aug 04 19:00:08.310861 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:00:08.310838    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca52852e-7dee-46f0-a413-6f14edf620cd-model-cache\") pod \"prestop-hook-test-kserve-765cd5b55b-tsq9t\" (UID: \"ca52852e-7dee-46f0-a413-6f14edf620cd\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-765cd5b55b-tsq9t"
Aug 04 19:00:08.310992 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:00:08.310949    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca52852e-7dee-46f0-a413-6f14edf620cd-home\") pod \"prestop-hook-test-kserve-765cd5b55b-tsq9t\" (UID: \"ca52852e-7dee-46f0-a413-6f14edf620cd\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-765cd5b55b-tsq9t"
Aug 04 19:00:08.311121 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:00:08.311018    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ca52852e-7dee-46f0-a413-6f14edf620cd-cabundle-cert\") pod \"prestop-hook-test-kserve-765cd5b55b-tsq9t\" (UID: \"ca52852e-7dee-46f0-a413-6f14edf620cd\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-765cd5b55b-tsq9t"
Aug 04 19:00:08.312837 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:00:08.312810    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca52852e-7dee-46f0-a413-6f14edf620cd-dshm\") pod \"prestop-hook-test-kserve-765cd5b55b-tsq9t\" (UID: \"ca52852e-7dee-46f0-a413-6f14edf620cd\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-765cd5b55b-tsq9t"
Aug 04 19:00:08.312983 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:00:08.312965    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca52852e-7dee-46f0-a413-6f14edf620cd-tls-certs\") pod \"prestop-hook-test-kserve-765cd5b55b-tsq9t\" (UID: \"ca52852e-7dee-46f0-a413-6f14edf620cd\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-765cd5b55b-tsq9t"
Aug 04 19:00:08.319655 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:00:08.319631    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjxxf\" (UniqueName: \"kubernetes.io/projected/ca52852e-7dee-46f0-a413-6f14edf620cd-kube-api-access-tjxxf\") pod \"prestop-hook-test-kserve-765cd5b55b-tsq9t\" (UID: \"ca52852e-7dee-46f0-a413-6f14edf620cd\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-765cd5b55b-tsq9t"
Aug 04 19:00:08.405738 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:00:08.405668    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-765cd5b55b-tsq9t"
Aug 04 19:00:08.550277 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:00:08.550251    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-765cd5b55b-tsq9t"]
Aug 04 19:00:08.552148 ip-10-0-135-124 kubenswrapper[2583]: W0804 19:00:08.552122    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca52852e_7dee_46f0_a413_6f14edf620cd.slice/crio-8eac31fb100c5cd5bf5eb2efe3852c69a041ef7dda8582952fb4b28be7916057 WatchSource:0}: Error finding container 8eac31fb100c5cd5bf5eb2efe3852c69a041ef7dda8582952fb4b28be7916057: Status 404 returned error can't find the container with id 8eac31fb100c5cd5bf5eb2efe3852c69a041ef7dda8582952fb4b28be7916057
Aug 04 19:00:09.014225 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:00:09.014170    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-765cd5b55b-tsq9t" event={"ID":"ca52852e-7dee-46f0-a413-6f14edf620cd","Type":"ContainerStarted","Data":"19e785ad7cafb7c273c2909381464123bf893c83208b2da03d71f79d4483cabc"}
Aug 04 19:00:09.014421 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:00:09.014247    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-765cd5b55b-tsq9t" event={"ID":"ca52852e-7dee-46f0-a413-6f14edf620cd","Type":"ContainerStarted","Data":"8eac31fb100c5cd5bf5eb2efe3852c69a041ef7dda8582952fb4b28be7916057"}
Aug 04 19:00:11.025987 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:00:11.025911    2583 generic.go:356] "Generic (PLEG): container finished" podID="ca52852e-7dee-46f0-a413-6f14edf620cd" containerID="19e785ad7cafb7c273c2909381464123bf893c83208b2da03d71f79d4483cabc" exitCode=0
Aug 04 19:00:11.026329 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:00:11.025982    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-765cd5b55b-tsq9t" event={"ID":"ca52852e-7dee-46f0-a413-6f14edf620cd","Type":"ContainerDied","Data":"19e785ad7cafb7c273c2909381464123bf893c83208b2da03d71f79d4483cabc"}
Aug 04 19:00:12.035035 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:00:12.035000    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-765cd5b55b-tsq9t" event={"ID":"ca52852e-7dee-46f0-a413-6f14edf620cd","Type":"ContainerStarted","Data":"55ff9575fa2875fda616ddff2a27dc0d9649806539eaa1bae8ea48e9a056ed68"}
Aug 04 19:00:12.060484 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:00:12.060409    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-765cd5b55b-tsq9t" podStartSLOduration=1.915262143 podStartE2EDuration="4.060388646s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.145126503s" isStatefulPod=false podCreationTimestamp="2026-08-04 19:00:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:00:12.058226182 +0000 UTC m=+3625.639444759" watchObservedRunningTime="2026-08-04 19:00:12.060388646 +0000 UTC m=+3625.641607225"
Aug 04 19:00:12.406515 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:00:12.406364    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-765cd5b55b-tsq9t"
Aug 04 19:00:14.099284 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:00:14.099247    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-759d7d7b7c-jw68j"]
Aug 04 19:00:18.406746 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:00:18.406704    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-765cd5b55b-tsq9t"
Aug 04 19:00:18.408405 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:00:18.408371    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-765cd5b55b-tsq9t" podUID="ca52852e-7dee-46f0-a413-6f14edf620cd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/health\": dial tcp 10.132.0.85:8000: connect: connection refused"
Aug 04 19:00:26.399155 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:00:26.399119    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-765cd5b55b-tsq9t"]
Aug 04 19:00:29.121927 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:00:29.121879    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-759d7d7b7c-jw68j" podUID="b849d68f-a68e-4a78-809b-540472fbc43b" containerName="main" containerID="cri-o://733c7d538f3cbfae3bcb85a2c30d8095706da61874a1a610d0c58904645204a8" gracePeriod=45
Aug 04 19:00:29.869238 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:00:29.869209    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-759d7d7b7c-jw68j"
Aug 04 19:00:30.012234 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:00:30.012207    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b849d68f-a68e-4a78-809b-540472fbc43b-model-cache\") pod \"b849d68f-a68e-4a78-809b-540472fbc43b\" (UID: \"b849d68f-a68e-4a78-809b-540472fbc43b\") "
Aug 04 19:00:30.012400 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:00:30.012272    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gv8nh\" (UniqueName: \"kubernetes.io/projected/b849d68f-a68e-4a78-809b-540472fbc43b-kube-api-access-gv8nh\") pod \"b849d68f-a68e-4a78-809b-540472fbc43b\" (UID: \"b849d68f-a68e-4a78-809b-540472fbc43b\") "
Aug 04 19:00:30.012400 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:00:30.012296    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b849d68f-a68e-4a78-809b-540472fbc43b-tls-certs\") pod \"b849d68f-a68e-4a78-809b-540472fbc43b\" (UID: \"b849d68f-a68e-4a78-809b-540472fbc43b\") "
Aug 04 19:00:30.012400 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:00:30.012311    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b849d68f-a68e-4a78-809b-540472fbc43b-tmp-dir\") pod \"b849d68f-a68e-4a78-809b-540472fbc43b\" (UID: \"b849d68f-a68e-4a78-809b-540472fbc43b\") "
Aug 04 19:00:30.012400 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:00:30.012332    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b849d68f-a68e-4a78-809b-540472fbc43b-cabundle-cert\") pod \"b849d68f-a68e-4a78-809b-540472fbc43b\" (UID: \"b849d68f-a68e-4a78-809b-540472fbc43b\") "
Aug 04 19:00:30.012400 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:00:30.012351    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b849d68f-a68e-4a78-809b-540472fbc43b-dshm\") pod \"b849d68f-a68e-4a78-809b-540472fbc43b\" (UID: \"b849d68f-a68e-4a78-809b-540472fbc43b\") "
Aug 04 19:00:30.012400 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:00:30.012379    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b849d68f-a68e-4a78-809b-540472fbc43b-home\") pod \"b849d68f-a68e-4a78-809b-540472fbc43b\" (UID: \"b849d68f-a68e-4a78-809b-540472fbc43b\") "
Aug 04 19:00:30.012793 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:00:30.012462    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b849d68f-a68e-4a78-809b-540472fbc43b-kserve-provision-location\") pod \"b849d68f-a68e-4a78-809b-540472fbc43b\" (UID: \"b849d68f-a68e-4a78-809b-540472fbc43b\") "
Aug 04 19:00:30.012793 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:00:30.012546    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b849d68f-a68e-4a78-809b-540472fbc43b-model-cache" (OuterVolumeSpecName: "model-cache") pod "b849d68f-a68e-4a78-809b-540472fbc43b" (UID: "b849d68f-a68e-4a78-809b-540472fbc43b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 19:00:30.012793 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:00:30.012746    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b849d68f-a68e-4a78-809b-540472fbc43b-model-cache\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 19:00:30.013092 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:00:30.013067    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b849d68f-a68e-4a78-809b-540472fbc43b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "b849d68f-a68e-4a78-809b-540472fbc43b" (UID: "b849d68f-a68e-4a78-809b-540472fbc43b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 19:00:30.013410 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:00:30.013385    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b849d68f-a68e-4a78-809b-540472fbc43b-home" (OuterVolumeSpecName: "home") pod "b849d68f-a68e-4a78-809b-540472fbc43b" (UID: "b849d68f-a68e-4a78-809b-540472fbc43b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 19:00:30.014868 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:00:30.014839    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b849d68f-a68e-4a78-809b-540472fbc43b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b849d68f-a68e-4a78-809b-540472fbc43b" (UID: "b849d68f-a68e-4a78-809b-540472fbc43b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 19:00:30.014973 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:00:30.014913    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b849d68f-a68e-4a78-809b-540472fbc43b-kube-api-access-gv8nh" (OuterVolumeSpecName: "kube-api-access-gv8nh") pod "b849d68f-a68e-4a78-809b-540472fbc43b" (UID: "b849d68f-a68e-4a78-809b-540472fbc43b"). InnerVolumeSpecName "kube-api-access-gv8nh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 19:00:30.015021 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:00:30.014969    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b849d68f-a68e-4a78-809b-540472fbc43b-dshm" (OuterVolumeSpecName: "dshm") pod "b849d68f-a68e-4a78-809b-540472fbc43b" (UID: "b849d68f-a68e-4a78-809b-540472fbc43b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 19:00:30.024429 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:00:30.024401    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b849d68f-a68e-4a78-809b-540472fbc43b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b849d68f-a68e-4a78-809b-540472fbc43b" (UID: "b849d68f-a68e-4a78-809b-540472fbc43b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 19:00:30.043751 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:00:30.043724    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b849d68f-a68e-4a78-809b-540472fbc43b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b849d68f-a68e-4a78-809b-540472fbc43b" (UID: "b849d68f-a68e-4a78-809b-540472fbc43b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 19:00:30.113191 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:00:30.113162    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gv8nh\" (UniqueName: \"kubernetes.io/projected/b849d68f-a68e-4a78-809b-540472fbc43b-kube-api-access-gv8nh\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 19:00:30.113191 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:00:30.113187    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b849d68f-a68e-4a78-809b-540472fbc43b-tls-certs\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 19:00:30.113191 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:00:30.113196    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b849d68f-a68e-4a78-809b-540472fbc43b-tmp-dir\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 19:00:30.113389 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:00:30.113204    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b849d68f-a68e-4a78-809b-540472fbc43b-cabundle-cert\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 19:00:30.113389 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:00:30.113212    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b849d68f-a68e-4a78-809b-540472fbc43b-dshm\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 19:00:30.113389 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:00:30.113219    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b849d68f-a68e-4a78-809b-540472fbc43b-home\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 19:00:30.113389 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:00:30.113226    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b849d68f-a68e-4a78-809b-540472fbc43b-kserve-provision-location\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 19:00:30.162529 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:00:30.162493    2583 generic.go:356] "Generic (PLEG): container finished" podID="b849d68f-a68e-4a78-809b-540472fbc43b" containerID="733c7d538f3cbfae3bcb85a2c30d8095706da61874a1a610d0c58904645204a8" exitCode=0
Aug 04 19:00:30.162893 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:00:30.162543    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-759d7d7b7c-jw68j"
Aug 04 19:00:30.162893 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:00:30.162561    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-759d7d7b7c-jw68j" event={"ID":"b849d68f-a68e-4a78-809b-540472fbc43b","Type":"ContainerDied","Data":"733c7d538f3cbfae3bcb85a2c30d8095706da61874a1a610d0c58904645204a8"}
Aug 04 19:00:30.162893 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:00:30.162623    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-759d7d7b7c-jw68j" event={"ID":"b849d68f-a68e-4a78-809b-540472fbc43b","Type":"ContainerDied","Data":"84c585bd0c1f89f08f8a7157eeb0c551f4b9c4f79b9ba4042b0b1714ac3913e4"}
Aug 04 19:00:30.162893 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:00:30.162646    2583 scope.go:117] "RemoveContainer" containerID="733c7d538f3cbfae3bcb85a2c30d8095706da61874a1a610d0c58904645204a8"
Aug 04 19:00:30.173257 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:00:30.173238    2583 scope.go:117] "RemoveContainer" containerID="2c8324478a379b7e4ffdffd3b6247229ca7ea297c57a579f96a49f8fa4d36130"
Aug 04 19:00:30.182789 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:00:30.182771    2583 scope.go:117] "RemoveContainer" containerID="733c7d538f3cbfae3bcb85a2c30d8095706da61874a1a610d0c58904645204a8"
Aug 04 19:00:30.183062 ip-10-0-135-124 kubenswrapper[2583]: E0804 19:00:30.183045    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"733c7d538f3cbfae3bcb85a2c30d8095706da61874a1a610d0c58904645204a8\": container with ID starting with 733c7d538f3cbfae3bcb85a2c30d8095706da61874a1a610d0c58904645204a8 not found: ID does not exist" containerID="733c7d538f3cbfae3bcb85a2c30d8095706da61874a1a610d0c58904645204a8"
Aug 04 19:00:30.183109 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:00:30.183070    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"733c7d538f3cbfae3bcb85a2c30d8095706da61874a1a610d0c58904645204a8"} err="failed to get container status \"733c7d538f3cbfae3bcb85a2c30d8095706da61874a1a610d0c58904645204a8\": rpc error: code = NotFound desc = could not find container \"733c7d538f3cbfae3bcb85a2c30d8095706da61874a1a610d0c58904645204a8\": container with ID starting with 733c7d538f3cbfae3bcb85a2c30d8095706da61874a1a610d0c58904645204a8 not found: ID does not exist"
Aug 04 19:00:30.183109 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:00:30.183086    2583 scope.go:117] "RemoveContainer" containerID="2c8324478a379b7e4ffdffd3b6247229ca7ea297c57a579f96a49f8fa4d36130"
Aug 04 19:00:30.183338 ip-10-0-135-124 kubenswrapper[2583]: E0804 19:00:30.183319    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c8324478a379b7e4ffdffd3b6247229ca7ea297c57a579f96a49f8fa4d36130\": container with ID starting with 2c8324478a379b7e4ffdffd3b6247229ca7ea297c57a579f96a49f8fa4d36130 not found: ID does not exist" containerID="2c8324478a379b7e4ffdffd3b6247229ca7ea297c57a579f96a49f8fa4d36130"
Aug 04 19:00:30.183383 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:00:30.183344    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c8324478a379b7e4ffdffd3b6247229ca7ea297c57a579f96a49f8fa4d36130"} err="failed to get container status \"2c8324478a379b7e4ffdffd3b6247229ca7ea297c57a579f96a49f8fa4d36130\": rpc error: code = NotFound desc = could not find container \"2c8324478a379b7e4ffdffd3b6247229ca7ea297c57a579f96a49f8fa4d36130\": container with ID starting with 2c8324478a379b7e4ffdffd3b6247229ca7ea297c57a579f96a49f8fa4d36130 not found: ID does not exist"
Aug 04 19:00:30.188996 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:00:30.188970    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-759d7d7b7c-jw68j"]
Aug 04 19:00:30.195941 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:00:30.195920    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-759d7d7b7c-jw68j"]
Aug 04 19:00:30.942432 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:00:30.942399    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b849d68f-a68e-4a78-809b-540472fbc43b" path="/var/lib/kubelet/pods/b849d68f-a68e-4a78-809b-540472fbc43b/volumes"
Aug 04 19:00:34.852934 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:00:34.852837    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-759d7d7b7c-jw68j" podUID="b849d68f-a68e-4a78-809b-540472fbc43b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/health\": context deadline exceeded"
Aug 04 19:00:41.423188 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:00:41.423124    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-765cd5b55b-tsq9t" podUID="ca52852e-7dee-46f0-a413-6f14edf620cd" containerName="main" containerID="cri-o://55ff9575fa2875fda616ddff2a27dc0d9649806539eaa1bae8ea48e9a056ed68" gracePeriod=45
Aug 04 19:01:27.074418 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:01:27.074388    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-prestop-hook-22727e61_prestop-hook-test-kserve-765cd5b55b-tsq9t_ca52852e-7dee-46f0-a413-6f14edf620cd/main/0.log"
Aug 04 19:01:27.077065 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:01:27.077044    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-765cd5b55b-tsq9t"
Aug 04 19:01:27.214224 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:01:27.214128    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ca52852e-7dee-46f0-a413-6f14edf620cd-kserve-provision-location\") pod \"ca52852e-7dee-46f0-a413-6f14edf620cd\" (UID: \"ca52852e-7dee-46f0-a413-6f14edf620cd\") "
Aug 04 19:01:27.214224 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:01:27.214169    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tjxxf\" (UniqueName: \"kubernetes.io/projected/ca52852e-7dee-46f0-a413-6f14edf620cd-kube-api-access-tjxxf\") pod \"ca52852e-7dee-46f0-a413-6f14edf620cd\" (UID: \"ca52852e-7dee-46f0-a413-6f14edf620cd\") "
Aug 04 19:01:27.214224 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:01:27.214194    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca52852e-7dee-46f0-a413-6f14edf620cd-dshm\") pod \"ca52852e-7dee-46f0-a413-6f14edf620cd\" (UID: \"ca52852e-7dee-46f0-a413-6f14edf620cd\") "
Aug 04 19:01:27.214224 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:01:27.214211    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ca52852e-7dee-46f0-a413-6f14edf620cd-cabundle-cert\") pod \"ca52852e-7dee-46f0-a413-6f14edf620cd\" (UID: \"ca52852e-7dee-46f0-a413-6f14edf620cd\") "
Aug 04 19:01:27.214596 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:01:27.214257    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca52852e-7dee-46f0-a413-6f14edf620cd-tls-certs\") pod \"ca52852e-7dee-46f0-a413-6f14edf620cd\" (UID: \"ca52852e-7dee-46f0-a413-6f14edf620cd\") "
Aug 04 19:01:27.214596 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:01:27.214308    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca52852e-7dee-46f0-a413-6f14edf620cd-model-cache\") pod \"ca52852e-7dee-46f0-a413-6f14edf620cd\" (UID: \"ca52852e-7dee-46f0-a413-6f14edf620cd\") "
Aug 04 19:01:27.214596 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:01:27.214360    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca52852e-7dee-46f0-a413-6f14edf620cd-home\") pod \"ca52852e-7dee-46f0-a413-6f14edf620cd\" (UID: \"ca52852e-7dee-46f0-a413-6f14edf620cd\") "
Aug 04 19:01:27.214596 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:01:27.214430    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca52852e-7dee-46f0-a413-6f14edf620cd-tmp-dir\") pod \"ca52852e-7dee-46f0-a413-6f14edf620cd\" (UID: \"ca52852e-7dee-46f0-a413-6f14edf620cd\") "
Aug 04 19:01:27.214802 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:01:27.214670    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca52852e-7dee-46f0-a413-6f14edf620cd-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ca52852e-7dee-46f0-a413-6f14edf620cd" (UID: "ca52852e-7dee-46f0-a413-6f14edf620cd"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 19:01:27.214802 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:01:27.214670    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca52852e-7dee-46f0-a413-6f14edf620cd-model-cache" (OuterVolumeSpecName: "model-cache") pod "ca52852e-7dee-46f0-a413-6f14edf620cd" (UID: "ca52852e-7dee-46f0-a413-6f14edf620cd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 19:01:27.215021 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:01:27.214991    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca52852e-7dee-46f0-a413-6f14edf620cd-home" (OuterVolumeSpecName: "home") pod "ca52852e-7dee-46f0-a413-6f14edf620cd" (UID: "ca52852e-7dee-46f0-a413-6f14edf620cd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 19:01:27.215157 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:01:27.215138    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca52852e-7dee-46f0-a413-6f14edf620cd-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ca52852e-7dee-46f0-a413-6f14edf620cd" (UID: "ca52852e-7dee-46f0-a413-6f14edf620cd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 19:01:27.216973 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:01:27.216940    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca52852e-7dee-46f0-a413-6f14edf620cd-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ca52852e-7dee-46f0-a413-6f14edf620cd" (UID: "ca52852e-7dee-46f0-a413-6f14edf620cd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 19:01:27.217218 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:01:27.217195    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca52852e-7dee-46f0-a413-6f14edf620cd-dshm" (OuterVolumeSpecName: "dshm") pod "ca52852e-7dee-46f0-a413-6f14edf620cd" (UID: "ca52852e-7dee-46f0-a413-6f14edf620cd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 19:01:27.217283 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:01:27.217214    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca52852e-7dee-46f0-a413-6f14edf620cd-kube-api-access-tjxxf" (OuterVolumeSpecName: "kube-api-access-tjxxf") pod "ca52852e-7dee-46f0-a413-6f14edf620cd" (UID: "ca52852e-7dee-46f0-a413-6f14edf620cd"). InnerVolumeSpecName "kube-api-access-tjxxf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 19:01:27.236583 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:01:27.236531    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca52852e-7dee-46f0-a413-6f14edf620cd-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ca52852e-7dee-46f0-a413-6f14edf620cd" (UID: "ca52852e-7dee-46f0-a413-6f14edf620cd"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 19:01:27.315401 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:01:27.315369    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca52852e-7dee-46f0-a413-6f14edf620cd-home\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 19:01:27.315401 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:01:27.315397    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca52852e-7dee-46f0-a413-6f14edf620cd-tmp-dir\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 19:01:27.315401 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:01:27.315408    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ca52852e-7dee-46f0-a413-6f14edf620cd-kserve-provision-location\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 19:01:27.315661 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:01:27.315417    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tjxxf\" (UniqueName: \"kubernetes.io/projected/ca52852e-7dee-46f0-a413-6f14edf620cd-kube-api-access-tjxxf\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 19:01:27.315661 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:01:27.315426    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca52852e-7dee-46f0-a413-6f14edf620cd-dshm\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 19:01:27.315661 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:01:27.315435    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ca52852e-7dee-46f0-a413-6f14edf620cd-cabundle-cert\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 19:01:27.315661 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:01:27.315443    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca52852e-7dee-46f0-a413-6f14edf620cd-tls-certs\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 19:01:27.315661 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:01:27.315450    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca52852e-7dee-46f0-a413-6f14edf620cd-model-cache\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 19:01:27.495375 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:01:27.495298    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-prestop-hook-22727e61_prestop-hook-test-kserve-765cd5b55b-tsq9t_ca52852e-7dee-46f0-a413-6f14edf620cd/main/0.log"
Aug 04 19:01:27.495679 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:01:27.495656    2583 generic.go:356] "Generic (PLEG): container finished" podID="ca52852e-7dee-46f0-a413-6f14edf620cd" containerID="55ff9575fa2875fda616ddff2a27dc0d9649806539eaa1bae8ea48e9a056ed68" exitCode=137
Aug 04 19:01:27.495774 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:01:27.495732    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-765cd5b55b-tsq9t"
Aug 04 19:01:27.495835 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:01:27.495737    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-765cd5b55b-tsq9t" event={"ID":"ca52852e-7dee-46f0-a413-6f14edf620cd","Type":"ContainerDied","Data":"55ff9575fa2875fda616ddff2a27dc0d9649806539eaa1bae8ea48e9a056ed68"}
Aug 04 19:01:27.495894 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:01:27.495852    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-765cd5b55b-tsq9t" event={"ID":"ca52852e-7dee-46f0-a413-6f14edf620cd","Type":"ContainerDied","Data":"8eac31fb100c5cd5bf5eb2efe3852c69a041ef7dda8582952fb4b28be7916057"}
Aug 04 19:01:27.495894 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:01:27.495875    2583 scope.go:117] "RemoveContainer" containerID="55ff9575fa2875fda616ddff2a27dc0d9649806539eaa1bae8ea48e9a056ed68"
Aug 04 19:01:27.505411 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:01:27.505388    2583 scope.go:117] "RemoveContainer" containerID="19e785ad7cafb7c273c2909381464123bf893c83208b2da03d71f79d4483cabc"
Aug 04 19:01:27.513437 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:01:27.513421    2583 scope.go:117] "RemoveContainer" containerID="55ff9575fa2875fda616ddff2a27dc0d9649806539eaa1bae8ea48e9a056ed68"
Aug 04 19:01:27.513710 ip-10-0-135-124 kubenswrapper[2583]: E0804 19:01:27.513690    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55ff9575fa2875fda616ddff2a27dc0d9649806539eaa1bae8ea48e9a056ed68\": container with ID starting with 55ff9575fa2875fda616ddff2a27dc0d9649806539eaa1bae8ea48e9a056ed68 not found: ID does not exist" containerID="55ff9575fa2875fda616ddff2a27dc0d9649806539eaa1bae8ea48e9a056ed68"
Aug 04 19:01:27.513767 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:01:27.513716    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55ff9575fa2875fda616ddff2a27dc0d9649806539eaa1bae8ea48e9a056ed68"} err="failed to get container status \"55ff9575fa2875fda616ddff2a27dc0d9649806539eaa1bae8ea48e9a056ed68\": rpc error: code = NotFound desc = could not find container \"55ff9575fa2875fda616ddff2a27dc0d9649806539eaa1bae8ea48e9a056ed68\": container with ID starting with 55ff9575fa2875fda616ddff2a27dc0d9649806539eaa1bae8ea48e9a056ed68 not found: ID does not exist"
Aug 04 19:01:27.513767 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:01:27.513732    2583 scope.go:117] "RemoveContainer" containerID="19e785ad7cafb7c273c2909381464123bf893c83208b2da03d71f79d4483cabc"
Aug 04 19:01:27.513944 ip-10-0-135-124 kubenswrapper[2583]: E0804 19:01:27.513926    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19e785ad7cafb7c273c2909381464123bf893c83208b2da03d71f79d4483cabc\": container with ID starting with 19e785ad7cafb7c273c2909381464123bf893c83208b2da03d71f79d4483cabc not found: ID does not exist" containerID="19e785ad7cafb7c273c2909381464123bf893c83208b2da03d71f79d4483cabc"
Aug 04 19:01:27.514002 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:01:27.513957    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19e785ad7cafb7c273c2909381464123bf893c83208b2da03d71f79d4483cabc"} err="failed to get container status \"19e785ad7cafb7c273c2909381464123bf893c83208b2da03d71f79d4483cabc\": rpc error: code = NotFound desc = could not find container \"19e785ad7cafb7c273c2909381464123bf893c83208b2da03d71f79d4483cabc\": container with ID starting with 19e785ad7cafb7c273c2909381464123bf893c83208b2da03d71f79d4483cabc not found: ID does not exist"
Aug 04 19:01:27.522903 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:01:27.522875    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-765cd5b55b-tsq9t"]
Aug 04 19:01:27.526445 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:01:27.526418    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-765cd5b55b-tsq9t"]
Aug 04 19:01:28.942007 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:01:28.941969    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ca52852e-7dee-46f0-a413-6f14edf620cd" path="/var/lib/kubelet/pods/ca52852e-7dee-46f0-a413-6f14edf620cd/volumes"
Aug 04 19:01:37.402500 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:01:37.402449    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-tz7r5"]
Aug 04 19:01:37.402981 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:01:37.402812    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b849d68f-a68e-4a78-809b-540472fbc43b" containerName="storage-initializer"
Aug 04 19:01:37.402981 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:01:37.402824    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="b849d68f-a68e-4a78-809b-540472fbc43b" containerName="storage-initializer"
Aug 04 19:01:37.402981 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:01:37.402841    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca52852e-7dee-46f0-a413-6f14edf620cd" containerName="main"
Aug 04 19:01:37.402981 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:01:37.402847    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca52852e-7dee-46f0-a413-6f14edf620cd" containerName="main"
Aug 04 19:01:37.402981 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:01:37.402855    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b849d68f-a68e-4a78-809b-540472fbc43b" containerName="main"
Aug 04 19:01:37.402981 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:01:37.402861    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="b849d68f-a68e-4a78-809b-540472fbc43b" containerName="main"
Aug 04 19:01:37.402981 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:01:37.402871    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca52852e-7dee-46f0-a413-6f14edf620cd" containerName="storage-initializer"
Aug 04 19:01:37.402981 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:01:37.402877    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca52852e-7dee-46f0-a413-6f14edf620cd" containerName="storage-initializer"
Aug 04 19:01:37.402981 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:01:37.402928    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca52852e-7dee-46f0-a413-6f14edf620cd" containerName="main"
Aug 04 19:01:37.402981 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:01:37.402936    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="b849d68f-a68e-4a78-809b-540472fbc43b" containerName="main"
Aug 04 19:01:37.405801 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:01:37.405783    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-tz7r5"
Aug 04 19:01:37.408477 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:01:37.408448    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"kube-root-ca.crt\""
Aug 04 19:01:37.408621 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:01:37.408522    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"openshift-service-ca.crt\""
Aug 04 19:01:37.409770 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:01:37.409751    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"default-dockercfg-lpzq8\""
Aug 04 19:01:37.409832 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:01:37.409767    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"rolling-upgrade-test-kserve-self-signed-certs\""
Aug 04 19:01:37.423909 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:01:37.423881    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-tz7r5"]
Aug 04 19:01:37.500851 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:01:37.500810    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/23de94ac-778e-48e1-a9bb-44860a56dfb5-tmp-dir\") pod \"rolling-upgrade-test-kserve-74b79d7f77-tz7r5\" (UID: \"23de94ac-778e-48e1-a9bb-44860a56dfb5\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-tz7r5"
Aug 04 19:01:37.500851 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:01:37.500853    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23de94ac-778e-48e1-a9bb-44860a56dfb5-tls-certs\") pod \"rolling-upgrade-test-kserve-74b79d7f77-tz7r5\" (UID: \"23de94ac-778e-48e1-a9bb-44860a56dfb5\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-tz7r5"
Aug 04 19:01:37.501066 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:01:37.500917    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8s86\" (UniqueName: \"kubernetes.io/projected/23de94ac-778e-48e1-a9bb-44860a56dfb5-kube-api-access-c8s86\") pod \"rolling-upgrade-test-kserve-74b79d7f77-tz7r5\" (UID: \"23de94ac-778e-48e1-a9bb-44860a56dfb5\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-tz7r5"
Aug 04 19:01:37.501066 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:01:37.500956    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/23de94ac-778e-48e1-a9bb-44860a56dfb5-model-cache\") pod \"rolling-upgrade-test-kserve-74b79d7f77-tz7r5\" (UID: \"23de94ac-778e-48e1-a9bb-44860a56dfb5\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-tz7r5"
Aug 04 19:01:37.501066 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:01:37.501007    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/23de94ac-778e-48e1-a9bb-44860a56dfb5-home\") pod \"rolling-upgrade-test-kserve-74b79d7f77-tz7r5\" (UID: \"23de94ac-778e-48e1-a9bb-44860a56dfb5\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-tz7r5"
Aug 04 19:01:37.501066 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:01:37.501042    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/23de94ac-778e-48e1-a9bb-44860a56dfb5-dshm\") pod \"rolling-upgrade-test-kserve-74b79d7f77-tz7r5\" (UID: \"23de94ac-778e-48e1-a9bb-44860a56dfb5\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-tz7r5"
Aug 04 19:01:37.601723 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:01:37.601684    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/23de94ac-778e-48e1-a9bb-44860a56dfb5-home\") pod \"rolling-upgrade-test-kserve-74b79d7f77-tz7r5\" (UID: \"23de94ac-778e-48e1-a9bb-44860a56dfb5\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-tz7r5"
Aug 04 19:01:37.601896 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:01:37.601734    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/23de94ac-778e-48e1-a9bb-44860a56dfb5-dshm\") pod \"rolling-upgrade-test-kserve-74b79d7f77-tz7r5\" (UID: \"23de94ac-778e-48e1-a9bb-44860a56dfb5\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-tz7r5"
Aug 04 19:01:37.601896 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:01:37.601772    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/23de94ac-778e-48e1-a9bb-44860a56dfb5-tmp-dir\") pod \"rolling-upgrade-test-kserve-74b79d7f77-tz7r5\" (UID: \"23de94ac-778e-48e1-a9bb-44860a56dfb5\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-tz7r5"
Aug 04 19:01:37.601896 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:01:37.601793    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23de94ac-778e-48e1-a9bb-44860a56dfb5-tls-certs\") pod \"rolling-upgrade-test-kserve-74b79d7f77-tz7r5\" (UID: \"23de94ac-778e-48e1-a9bb-44860a56dfb5\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-tz7r5"
Aug 04 19:01:37.601896 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:01:37.601825    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8s86\" (UniqueName: \"kubernetes.io/projected/23de94ac-778e-48e1-a9bb-44860a56dfb5-kube-api-access-c8s86\") pod \"rolling-upgrade-test-kserve-74b79d7f77-tz7r5\" (UID: \"23de94ac-778e-48e1-a9bb-44860a56dfb5\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-tz7r5"
Aug 04 19:01:37.602125 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:01:37.601980    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/23de94ac-778e-48e1-a9bb-44860a56dfb5-model-cache\") pod \"rolling-upgrade-test-kserve-74b79d7f77-tz7r5\" (UID: \"23de94ac-778e-48e1-a9bb-44860a56dfb5\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-tz7r5"
Aug 04 19:01:37.602185 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:01:37.602149    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/23de94ac-778e-48e1-a9bb-44860a56dfb5-home\") pod \"rolling-upgrade-test-kserve-74b79d7f77-tz7r5\" (UID: \"23de94ac-778e-48e1-a9bb-44860a56dfb5\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-tz7r5"
Aug 04 19:01:37.602241 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:01:37.602180    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/23de94ac-778e-48e1-a9bb-44860a56dfb5-tmp-dir\") pod \"rolling-upgrade-test-kserve-74b79d7f77-tz7r5\" (UID: \"23de94ac-778e-48e1-a9bb-44860a56dfb5\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-tz7r5"
Aug 04 19:01:37.602328 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:01:37.602308    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/23de94ac-778e-48e1-a9bb-44860a56dfb5-model-cache\") pod \"rolling-upgrade-test-kserve-74b79d7f77-tz7r5\" (UID: \"23de94ac-778e-48e1-a9bb-44860a56dfb5\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-tz7r5"
Aug 04 19:01:37.604240 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:01:37.604215    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/23de94ac-778e-48e1-a9bb-44860a56dfb5-dshm\") pod \"rolling-upgrade-test-kserve-74b79d7f77-tz7r5\" (UID: \"23de94ac-778e-48e1-a9bb-44860a56dfb5\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-tz7r5"
Aug 04 19:01:37.604420 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:01:37.604400    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23de94ac-778e-48e1-a9bb-44860a56dfb5-tls-certs\") pod \"rolling-upgrade-test-kserve-74b79d7f77-tz7r5\" (UID: \"23de94ac-778e-48e1-a9bb-44860a56dfb5\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-tz7r5"
Aug 04 19:01:37.609938 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:01:37.609908    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8s86\" (UniqueName: \"kubernetes.io/projected/23de94ac-778e-48e1-a9bb-44860a56dfb5-kube-api-access-c8s86\") pod \"rolling-upgrade-test-kserve-74b79d7f77-tz7r5\" (UID: \"23de94ac-778e-48e1-a9bb-44860a56dfb5\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-tz7r5"
Aug 04 19:01:37.719209 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:01:37.719155    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-tz7r5"
Aug 04 19:01:37.859731 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:01:37.859696    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-tz7r5"]
Aug 04 19:01:37.861611 ip-10-0-135-124 kubenswrapper[2583]: W0804 19:01:37.861580    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23de94ac_778e_48e1_a9bb_44860a56dfb5.slice/crio-4fb5499ce4f483700187382bac124faca65d613143fd3f21c4783982e04ff47a WatchSource:0}: Error finding container 4fb5499ce4f483700187382bac124faca65d613143fd3f21c4783982e04ff47a: Status 404 returned error can't find the container with id 4fb5499ce4f483700187382bac124faca65d613143fd3f21c4783982e04ff47a
Aug 04 19:01:38.555097 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:01:38.555058    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-tz7r5" event={"ID":"23de94ac-778e-48e1-a9bb-44860a56dfb5","Type":"ContainerStarted","Data":"1b1f37430506c88753e56dd639ad486fdc4b2f179177ff08c516ade33406cfcb"}
Aug 04 19:01:38.555515 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:01:38.555125    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-tz7r5" event={"ID":"23de94ac-778e-48e1-a9bb-44860a56dfb5","Type":"ContainerStarted","Data":"4fb5499ce4f483700187382bac124faca65d613143fd3f21c4783982e04ff47a"}
Aug 04 19:01:38.579321 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:01:38.579271    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-tz7r5" podStartSLOduration=1.579254025 podStartE2EDuration="1.579254025s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 19:01:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:01:38.577006553 +0000 UTC m=+3712.158225121" watchObservedRunningTime="2026-08-04 19:01:38.579254025 +0000 UTC m=+3712.160472602"
Aug 04 19:01:38.720097 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:01:38.720061    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-tz7r5"
Aug 04 19:01:47.719947 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:01:47.719906    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-tz7r5"
Aug 04 19:01:47.732851 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:01:47.732823    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-tz7r5"
Aug 04 19:01:48.618708 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:01:48.618677    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-tz7r5"
Aug 04 19:02:17.337703 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:02:17.337662    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-tz7r5"]
Aug 04 19:02:17.346694 ip-10-0-135-124 kubenswrapper[2583]: E0804 19:02:17.346652    2583 log.go:32] "ExecSync cmd from runtime service failed" err=<
Aug 04 19:02:17.346694 ip-10-0-135-124 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Aug 04 19:02:17.346694 ip-10-0-135-124 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Aug 04 19:02:17.346694 ip-10-0-135-124 kubenswrapper[2583]:  > containerID="1b1f37430506c88753e56dd639ad486fdc4b2f179177ff08c516ade33406cfcb" cmd=["/bin/sleep","15"]
Aug 04 19:02:17.346935 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:02:17.346708    2583 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-tz7r5" message="" err=<
Aug 04 19:02:17.346935 ip-10-0-135-124 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Aug 04 19:02:17.346935 ip-10-0-135-124 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Aug 04 19:02:17.346935 ip-10-0-135-124 kubenswrapper[2583]:  >
Aug 04 19:02:17.346935 ip-10-0-135-124 kubenswrapper[2583]: E0804 19:02:17.346726    2583 kuberuntime_container.go:755] "PreStop hook failed" err=<
Aug 04 19:02:17.346935 ip-10-0-135-124 kubenswrapper[2583]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Aug 04 19:02:17.346935 ip-10-0-135-124 kubenswrapper[2583]:         , stdout: , stderr: , exit code -1
Aug 04 19:02:17.346935 ip-10-0-135-124 kubenswrapper[2583]:  > pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-tz7r5" podUID="23de94ac-778e-48e1-a9bb-44860a56dfb5" containerName="main" containerID="cri-o://1b1f37430506c88753e56dd639ad486fdc4b2f179177ff08c516ade33406cfcb"
Aug 04 19:02:17.346935 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:02:17.346764    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-tz7r5" podUID="23de94ac-778e-48e1-a9bb-44860a56dfb5" containerName="main" containerID="cri-o://1b1f37430506c88753e56dd639ad486fdc4b2f179177ff08c516ade33406cfcb" gracePeriod=60
Aug 04 19:02:17.608741 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:02:17.608650    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-tz7r5" podUID="23de94ac-778e-48e1-a9bb-44860a56dfb5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/health\": dial tcp 10.132.0.86:8000: connect: connection refused"
Aug 04 19:02:17.700779 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:02:17.700747    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-tz7r5"
Aug 04 19:02:17.775917 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:02:17.775878    2583 generic.go:356] "Generic (PLEG): container finished" podID="23de94ac-778e-48e1-a9bb-44860a56dfb5" containerID="1b1f37430506c88753e56dd639ad486fdc4b2f179177ff08c516ade33406cfcb" exitCode=0
Aug 04 19:02:17.776113 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:02:17.775922    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-tz7r5" event={"ID":"23de94ac-778e-48e1-a9bb-44860a56dfb5","Type":"ContainerDied","Data":"1b1f37430506c88753e56dd639ad486fdc4b2f179177ff08c516ade33406cfcb"}
Aug 04 19:02:17.776113 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:02:17.775935    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-tz7r5"
Aug 04 19:02:17.776113 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:02:17.776007    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-tz7r5" event={"ID":"23de94ac-778e-48e1-a9bb-44860a56dfb5","Type":"ContainerDied","Data":"4fb5499ce4f483700187382bac124faca65d613143fd3f21c4783982e04ff47a"}
Aug 04 19:02:17.776113 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:02:17.776036    2583 scope.go:117] "RemoveContainer" containerID="1b1f37430506c88753e56dd639ad486fdc4b2f179177ff08c516ade33406cfcb"
Aug 04 19:02:17.785179 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:02:17.785160    2583 scope.go:117] "RemoveContainer" containerID="1b1f37430506c88753e56dd639ad486fdc4b2f179177ff08c516ade33406cfcb"
Aug 04 19:02:17.785453 ip-10-0-135-124 kubenswrapper[2583]: E0804 19:02:17.785432    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b1f37430506c88753e56dd639ad486fdc4b2f179177ff08c516ade33406cfcb\": container with ID starting with 1b1f37430506c88753e56dd639ad486fdc4b2f179177ff08c516ade33406cfcb not found: ID does not exist" containerID="1b1f37430506c88753e56dd639ad486fdc4b2f179177ff08c516ade33406cfcb"
Aug 04 19:02:17.785548 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:02:17.785490    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b1f37430506c88753e56dd639ad486fdc4b2f179177ff08c516ade33406cfcb"} err="failed to get container status \"1b1f37430506c88753e56dd639ad486fdc4b2f179177ff08c516ade33406cfcb\": rpc error: code = NotFound desc = could not find container \"1b1f37430506c88753e56dd639ad486fdc4b2f179177ff08c516ade33406cfcb\": container with ID starting with 1b1f37430506c88753e56dd639ad486fdc4b2f179177ff08c516ade33406cfcb not found: ID does not exist"
Aug 04 19:02:17.863845 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:02:17.863730    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/23de94ac-778e-48e1-a9bb-44860a56dfb5-dshm\") pod \"23de94ac-778e-48e1-a9bb-44860a56dfb5\" (UID: \"23de94ac-778e-48e1-a9bb-44860a56dfb5\") "
Aug 04 19:02:17.864050 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:02:17.863861    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23de94ac-778e-48e1-a9bb-44860a56dfb5-tls-certs\") pod \"23de94ac-778e-48e1-a9bb-44860a56dfb5\" (UID: \"23de94ac-778e-48e1-a9bb-44860a56dfb5\") "
Aug 04 19:02:17.864050 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:02:17.863883    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/23de94ac-778e-48e1-a9bb-44860a56dfb5-tmp-dir\") pod \"23de94ac-778e-48e1-a9bb-44860a56dfb5\" (UID: \"23de94ac-778e-48e1-a9bb-44860a56dfb5\") "
Aug 04 19:02:17.864050 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:02:17.863921    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/23de94ac-778e-48e1-a9bb-44860a56dfb5-model-cache\") pod \"23de94ac-778e-48e1-a9bb-44860a56dfb5\" (UID: \"23de94ac-778e-48e1-a9bb-44860a56dfb5\") "
Aug 04 19:02:17.864050 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:02:17.863941    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/23de94ac-778e-48e1-a9bb-44860a56dfb5-home\") pod \"23de94ac-778e-48e1-a9bb-44860a56dfb5\" (UID: \"23de94ac-778e-48e1-a9bb-44860a56dfb5\") "
Aug 04 19:02:17.864050 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:02:17.863983    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c8s86\" (UniqueName: \"kubernetes.io/projected/23de94ac-778e-48e1-a9bb-44860a56dfb5-kube-api-access-c8s86\") pod \"23de94ac-778e-48e1-a9bb-44860a56dfb5\" (UID: \"23de94ac-778e-48e1-a9bb-44860a56dfb5\") "
Aug 04 19:02:17.864333 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:02:17.864206    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23de94ac-778e-48e1-a9bb-44860a56dfb5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "23de94ac-778e-48e1-a9bb-44860a56dfb5" (UID: "23de94ac-778e-48e1-a9bb-44860a56dfb5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 19:02:17.864333 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:02:17.864246    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23de94ac-778e-48e1-a9bb-44860a56dfb5-home" (OuterVolumeSpecName: "home") pod "23de94ac-778e-48e1-a9bb-44860a56dfb5" (UID: "23de94ac-778e-48e1-a9bb-44860a56dfb5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 19:02:17.864333 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:02:17.864245    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23de94ac-778e-48e1-a9bb-44860a56dfb5-model-cache" (OuterVolumeSpecName: "model-cache") pod "23de94ac-778e-48e1-a9bb-44860a56dfb5" (UID: "23de94ac-778e-48e1-a9bb-44860a56dfb5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 19:02:17.866183 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:02:17.866151    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23de94ac-778e-48e1-a9bb-44860a56dfb5-dshm" (OuterVolumeSpecName: "dshm") pod "23de94ac-778e-48e1-a9bb-44860a56dfb5" (UID: "23de94ac-778e-48e1-a9bb-44860a56dfb5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 19:02:17.866295 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:02:17.866211    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23de94ac-778e-48e1-a9bb-44860a56dfb5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "23de94ac-778e-48e1-a9bb-44860a56dfb5" (UID: "23de94ac-778e-48e1-a9bb-44860a56dfb5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 19:02:17.866295 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:02:17.866281    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23de94ac-778e-48e1-a9bb-44860a56dfb5-kube-api-access-c8s86" (OuterVolumeSpecName: "kube-api-access-c8s86") pod "23de94ac-778e-48e1-a9bb-44860a56dfb5" (UID: "23de94ac-778e-48e1-a9bb-44860a56dfb5"). InnerVolumeSpecName "kube-api-access-c8s86". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 19:02:17.964973 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:02:17.964932    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/23de94ac-778e-48e1-a9bb-44860a56dfb5-tmp-dir\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 19:02:17.964973 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:02:17.964965    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/23de94ac-778e-48e1-a9bb-44860a56dfb5-model-cache\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 19:02:17.964973 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:02:17.964974    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/23de94ac-778e-48e1-a9bb-44860a56dfb5-home\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 19:02:17.965220 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:02:17.964985    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c8s86\" (UniqueName: \"kubernetes.io/projected/23de94ac-778e-48e1-a9bb-44860a56dfb5-kube-api-access-c8s86\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 19:02:17.965220 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:02:17.964995    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/23de94ac-778e-48e1-a9bb-44860a56dfb5-dshm\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 19:02:17.965220 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:02:17.965003    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23de94ac-778e-48e1-a9bb-44860a56dfb5-tls-certs\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 19:02:18.100660 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:02:18.100621    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-tz7r5"]
Aug 04 19:02:18.105016 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:02:18.104977    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-tz7r5"]
Aug 04 19:02:18.942934 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:02:18.942903    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="23de94ac-778e-48e1-a9bb-44860a56dfb5" path="/var/lib/kubelet/pods/23de94ac-778e-48e1-a9bb-44860a56dfb5/volumes"
Aug 04 19:04:19.527091 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:04:19.526995    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-86ccf66d9fbk9jk"]
Aug 04 19:04:19.527580 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:04:19.527531    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23de94ac-778e-48e1-a9bb-44860a56dfb5" containerName="main"
Aug 04 19:04:19.527580 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:04:19.527547    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="23de94ac-778e-48e1-a9bb-44860a56dfb5" containerName="main"
Aug 04 19:04:19.527681 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:04:19.527610    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="23de94ac-778e-48e1-a9bb-44860a56dfb5" containerName="main"
Aug 04 19:04:19.530735 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:04:19.530713    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-86ccf66d9fbk9jk"
Aug 04 19:04:19.533623 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:04:19.533577    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"odh-kserve-custom-ca-bundle\""
Aug 04 19:04:19.533623 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:04:19.533605    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"seaweedfs-s3-creds\""
Aug 04 19:04:19.533777 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:04:19.533653    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"openshift-service-ca.crt\""
Aug 04 19:04:19.534115 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:04:19.534088    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"kube-root-ca.crt\""
Aug 04 19:04:19.534227 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:04:19.534133    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"llmisvd10b93c8eba12f87c3a350a5cae2ee0c-kserve-self-signed-certs\""
Aug 04 19:04:19.534227 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:04:19.534173    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"default-dockercfg-rjp44\""
Aug 04 19:04:19.546570 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:04:19.546544    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-86ccf66d9fbk9jk"]
Aug 04 19:04:19.694006 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:04:19.693964    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b90a7b1a-cdf4-42cf-8dc7-3df9825dd91d-home\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-86ccf66d9fbk9jk\" (UID: \"b90a7b1a-cdf4-42cf-8dc7-3df9825dd91d\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-86ccf66d9fbk9jk"
Aug 04 19:04:19.694194 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:04:19.694017    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b90a7b1a-cdf4-42cf-8dc7-3df9825dd91d-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-86ccf66d9fbk9jk\" (UID: \"b90a7b1a-cdf4-42cf-8dc7-3df9825dd91d\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-86ccf66d9fbk9jk"
Aug 04 19:04:19.694194 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:04:19.694071    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b90a7b1a-cdf4-42cf-8dc7-3df9825dd91d-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-86ccf66d9fbk9jk\" (UID: \"b90a7b1a-cdf4-42cf-8dc7-3df9825dd91d\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-86ccf66d9fbk9jk"
Aug 04 19:04:19.694194 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:04:19.694102    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b90a7b1a-cdf4-42cf-8dc7-3df9825dd91d-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-86ccf66d9fbk9jk\" (UID: \"b90a7b1a-cdf4-42cf-8dc7-3df9825dd91d\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-86ccf66d9fbk9jk"
Aug 04 19:04:19.694194 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:04:19.694128    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztp5g\" (UniqueName: \"kubernetes.io/projected/b90a7b1a-cdf4-42cf-8dc7-3df9825dd91d-kube-api-access-ztp5g\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-86ccf66d9fbk9jk\" (UID: \"b90a7b1a-cdf4-42cf-8dc7-3df9825dd91d\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-86ccf66d9fbk9jk"
Aug 04 19:04:19.694194 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:04:19.694150    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b90a7b1a-cdf4-42cf-8dc7-3df9825dd91d-dshm\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-86ccf66d9fbk9jk\" (UID: \"b90a7b1a-cdf4-42cf-8dc7-3df9825dd91d\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-86ccf66d9fbk9jk"
Aug 04 19:04:19.694194 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:04:19.694188    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b90a7b1a-cdf4-42cf-8dc7-3df9825dd91d-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-86ccf66d9fbk9jk\" (UID: \"b90a7b1a-cdf4-42cf-8dc7-3df9825dd91d\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-86ccf66d9fbk9jk"
Aug 04 19:04:19.694423 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:04:19.694243    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b90a7b1a-cdf4-42cf-8dc7-3df9825dd91d-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-86ccf66d9fbk9jk\" (UID: \"b90a7b1a-cdf4-42cf-8dc7-3df9825dd91d\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-86ccf66d9fbk9jk"
Aug 04 19:04:19.795459 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:04:19.795365    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b90a7b1a-cdf4-42cf-8dc7-3df9825dd91d-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-86ccf66d9fbk9jk\" (UID: \"b90a7b1a-cdf4-42cf-8dc7-3df9825dd91d\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-86ccf66d9fbk9jk"
Aug 04 19:04:19.795459 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:04:19.795416    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b90a7b1a-cdf4-42cf-8dc7-3df9825dd91d-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-86ccf66d9fbk9jk\" (UID: \"b90a7b1a-cdf4-42cf-8dc7-3df9825dd91d\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-86ccf66d9fbk9jk"
Aug 04 19:04:19.795459 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:04:19.795444    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b90a7b1a-cdf4-42cf-8dc7-3df9825dd91d-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-86ccf66d9fbk9jk\" (UID: \"b90a7b1a-cdf4-42cf-8dc7-3df9825dd91d\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-86ccf66d9fbk9jk"
Aug 04 19:04:19.795765 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:04:19.795465    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztp5g\" (UniqueName: \"kubernetes.io/projected/b90a7b1a-cdf4-42cf-8dc7-3df9825dd91d-kube-api-access-ztp5g\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-86ccf66d9fbk9jk\" (UID: \"b90a7b1a-cdf4-42cf-8dc7-3df9825dd91d\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-86ccf66d9fbk9jk"
Aug 04 19:04:19.795765 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:04:19.795522    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b90a7b1a-cdf4-42cf-8dc7-3df9825dd91d-dshm\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-86ccf66d9fbk9jk\" (UID: \"b90a7b1a-cdf4-42cf-8dc7-3df9825dd91d\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-86ccf66d9fbk9jk"
Aug 04 19:04:19.795765 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:04:19.795569    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b90a7b1a-cdf4-42cf-8dc7-3df9825dd91d-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-86ccf66d9fbk9jk\" (UID: \"b90a7b1a-cdf4-42cf-8dc7-3df9825dd91d\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-86ccf66d9fbk9jk"
Aug 04 19:04:19.795765 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:04:19.795594    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b90a7b1a-cdf4-42cf-8dc7-3df9825dd91d-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-86ccf66d9fbk9jk\" (UID: \"b90a7b1a-cdf4-42cf-8dc7-3df9825dd91d\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-86ccf66d9fbk9jk"
Aug 04 19:04:19.795765 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:04:19.795638    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b90a7b1a-cdf4-42cf-8dc7-3df9825dd91d-home\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-86ccf66d9fbk9jk\" (UID: \"b90a7b1a-cdf4-42cf-8dc7-3df9825dd91d\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-86ccf66d9fbk9jk"
Aug 04 19:04:19.796035 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:04:19.795977    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b90a7b1a-cdf4-42cf-8dc7-3df9825dd91d-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-86ccf66d9fbk9jk\" (UID: \"b90a7b1a-cdf4-42cf-8dc7-3df9825dd91d\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-86ccf66d9fbk9jk"
Aug 04 19:04:19.796035 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:04:19.795989    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b90a7b1a-cdf4-42cf-8dc7-3df9825dd91d-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-86ccf66d9fbk9jk\" (UID: \"b90a7b1a-cdf4-42cf-8dc7-3df9825dd91d\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-86ccf66d9fbk9jk"
Aug 04 19:04:19.796155 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:04:19.796124    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b90a7b1a-cdf4-42cf-8dc7-3df9825dd91d-home\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-86ccf66d9fbk9jk\" (UID: \"b90a7b1a-cdf4-42cf-8dc7-3df9825dd91d\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-86ccf66d9fbk9jk"
Aug 04 19:04:19.796219 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:04:19.796156    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b90a7b1a-cdf4-42cf-8dc7-3df9825dd91d-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-86ccf66d9fbk9jk\" (UID: \"b90a7b1a-cdf4-42cf-8dc7-3df9825dd91d\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-86ccf66d9fbk9jk"
Aug 04 19:04:19.796433 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:04:19.796411    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b90a7b1a-cdf4-42cf-8dc7-3df9825dd91d-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-86ccf66d9fbk9jk\" (UID: \"b90a7b1a-cdf4-42cf-8dc7-3df9825dd91d\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-86ccf66d9fbk9jk"
Aug 04 19:04:19.797977 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:04:19.797955    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b90a7b1a-cdf4-42cf-8dc7-3df9825dd91d-dshm\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-86ccf66d9fbk9jk\" (UID: \"b90a7b1a-cdf4-42cf-8dc7-3df9825dd91d\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-86ccf66d9fbk9jk"
Aug 04 19:04:19.798084 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:04:19.798043    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b90a7b1a-cdf4-42cf-8dc7-3df9825dd91d-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-86ccf66d9fbk9jk\" (UID: \"b90a7b1a-cdf4-42cf-8dc7-3df9825dd91d\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-86ccf66d9fbk9jk"
Aug 04 19:04:19.803962 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:04:19.803936    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztp5g\" (UniqueName: \"kubernetes.io/projected/b90a7b1a-cdf4-42cf-8dc7-3df9825dd91d-kube-api-access-ztp5g\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-86ccf66d9fbk9jk\" (UID: \"b90a7b1a-cdf4-42cf-8dc7-3df9825dd91d\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-86ccf66d9fbk9jk"
Aug 04 19:04:19.844624 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:04:19.844587    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-86ccf66d9fbk9jk"
Aug 04 19:04:19.990870 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:04:19.990835    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-86ccf66d9fbk9jk"]
Aug 04 19:04:19.993008 ip-10-0-135-124 kubenswrapper[2583]: W0804 19:04:19.992978    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb90a7b1a_cdf4_42cf_8dc7_3df9825dd91d.slice/crio-830b8c9cffc8bcf2d1981d45057d04bd5c6c9989d8ed697624b3c2021a465bd4 WatchSource:0}: Error finding container 830b8c9cffc8bcf2d1981d45057d04bd5c6c9989d8ed697624b3c2021a465bd4: Status 404 returned error can't find the container with id 830b8c9cffc8bcf2d1981d45057d04bd5c6c9989d8ed697624b3c2021a465bd4
Aug 04 19:04:20.421299 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:04:20.421206    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-86ccf66d9fbk9jk" event={"ID":"b90a7b1a-cdf4-42cf-8dc7-3df9825dd91d","Type":"ContainerStarted","Data":"d14ba29b22f896b131dd9adb4bd0decfd43869a55f9ce8c46cd37bc236922565"}
Aug 04 19:04:20.421299 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:04:20.421274    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-86ccf66d9fbk9jk" event={"ID":"b90a7b1a-cdf4-42cf-8dc7-3df9825dd91d","Type":"ContainerStarted","Data":"830b8c9cffc8bcf2d1981d45057d04bd5c6c9989d8ed697624b3c2021a465bd4"}
Aug 04 19:04:22.433891 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:04:22.433857    2583 generic.go:356] "Generic (PLEG): container finished" podID="b90a7b1a-cdf4-42cf-8dc7-3df9825dd91d" containerID="d14ba29b22f896b131dd9adb4bd0decfd43869a55f9ce8c46cd37bc236922565" exitCode=0
Aug 04 19:04:22.434343 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:04:22.433927    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-86ccf66d9fbk9jk" event={"ID":"b90a7b1a-cdf4-42cf-8dc7-3df9825dd91d","Type":"ContainerDied","Data":"d14ba29b22f896b131dd9adb4bd0decfd43869a55f9ce8c46cd37bc236922565"}
Aug 04 19:04:23.441535 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:04:23.441497    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-86ccf66d9fbk9jk" event={"ID":"b90a7b1a-cdf4-42cf-8dc7-3df9825dd91d","Type":"ContainerStarted","Data":"bb239d0365fdecd8c6c4fd64067daa6e6e627f6f9f00bef4181f8dc329d3ab98"}
Aug 04 19:04:23.469318 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:04:23.469261    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-86ccf66d9fbk9jk" podStartSLOduration=2.333964973 podStartE2EDuration="4.469244517s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.135279544s" isStatefulPod=false podCreationTimestamp="2026-08-04 19:04:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:04:23.465840794 +0000 UTC m=+3877.047059374" watchObservedRunningTime="2026-08-04 19:04:23.469244517 +0000 UTC m=+3877.050463152"
Aug 04 19:04:23.845335 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:04:23.845244    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-86ccf66d9fbk9jk"
Aug 04 19:04:29.845549 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:04:29.845505    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-86ccf66d9fbk9jk"
Aug 04 19:04:29.847385 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:04:29.847350    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-86ccf66d9fbk9jk" podUID="b90a7b1a-cdf4-42cf-8dc7-3df9825dd91d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/health\": dial tcp 10.132.0.87:8000: connect: connection refused"
Aug 04 19:04:39.845946 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:04:39.845895    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-86ccf66d9fbk9jk" podUID="b90a7b1a-cdf4-42cf-8dc7-3df9825dd91d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/health\": dial tcp 10.132.0.87:8000: connect: connection refused"
Aug 04 19:04:47.302367 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:04:47.302253    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-wskfb_6157c2d3-2fd0-4544-88ad-434ac75f8c04/console-operator/1.log"
Aug 04 19:04:47.314982 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:04:47.314953    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-wskfb_6157c2d3-2fd0-4544-88ad-434ac75f8c04/console-operator/1.log"
Aug 04 19:04:49.845800 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:04:49.845758    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-86ccf66d9fbk9jk" podUID="b90a7b1a-cdf4-42cf-8dc7-3df9825dd91d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/health\": dial tcp 10.132.0.87:8000: connect: connection refused"
Aug 04 19:04:59.845505 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:04:59.845442    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-86ccf66d9fbk9jk" podUID="b90a7b1a-cdf4-42cf-8dc7-3df9825dd91d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/health\": dial tcp 10.132.0.87:8000: connect: connection refused"
Aug 04 19:05:09.845924 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:05:09.845878    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-86ccf66d9fbk9jk" podUID="b90a7b1a-cdf4-42cf-8dc7-3df9825dd91d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/health\": dial tcp 10.132.0.87:8000: connect: connection refused"
Aug 04 19:05:19.845524 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:05:19.845450    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-86ccf66d9fbk9jk" podUID="b90a7b1a-cdf4-42cf-8dc7-3df9825dd91d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/health\": dial tcp 10.132.0.87:8000: connect: connection refused"
Aug 04 19:05:29.845425 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:05:29.845376    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-86ccf66d9fbk9jk" podUID="b90a7b1a-cdf4-42cf-8dc7-3df9825dd91d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/health\": dial tcp 10.132.0.87:8000: connect: connection refused"
Aug 04 19:05:39.845463 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:05:39.845361    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-86ccf66d9fbk9jk" podUID="b90a7b1a-cdf4-42cf-8dc7-3df9825dd91d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/health\": dial tcp 10.132.0.87:8000: connect: connection refused"
Aug 04 19:05:49.845719 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:05:49.845667    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-86ccf66d9fbk9jk" podUID="b90a7b1a-cdf4-42cf-8dc7-3df9825dd91d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/health\": dial tcp 10.132.0.87:8000: connect: connection refused"
Aug 04 19:05:59.845975 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:05:59.845927    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-86ccf66d9fbk9jk" podUID="b90a7b1a-cdf4-42cf-8dc7-3df9825dd91d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/health\": dial tcp 10.132.0.87:8000: connect: connection refused"
Aug 04 19:06:09.845596 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:06:09.845542    2583 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-86ccf66d9fbk9jk" podUID="b90a7b1a-cdf4-42cf-8dc7-3df9825dd91d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/health\": dial tcp 10.132.0.87:8000: connect: connection refused"
Aug 04 19:06:19.856087 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:06:19.856044    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-86ccf66d9fbk9jk"
Aug 04 19:06:19.864231 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:06:19.864204    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-86ccf66d9fbk9jk"
Aug 04 19:06:24.254227 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:06:24.254189    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-86ccf66d9fbk9jk"]
Aug 04 19:06:39.277158 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:06:39.277121    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-86ccf66d9fbk9jk" podUID="b90a7b1a-cdf4-42cf-8dc7-3df9825dd91d" containerName="main" containerID="cri-o://bb239d0365fdecd8c6c4fd64067daa6e6e627f6f9f00bef4181f8dc329d3ab98" gracePeriod=45
Aug 04 19:06:39.923061 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:06:39.923037    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-86ccf66d9fbk9jk"
Aug 04 19:06:40.040808 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:06:40.040742    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b90a7b1a-cdf4-42cf-8dc7-3df9825dd91d-tls-certs\") pod \"b90a7b1a-cdf4-42cf-8dc7-3df9825dd91d\" (UID: \"b90a7b1a-cdf4-42cf-8dc7-3df9825dd91d\") "
Aug 04 19:06:40.040808 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:06:40.040776    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b90a7b1a-cdf4-42cf-8dc7-3df9825dd91d-kserve-provision-location\") pod \"b90a7b1a-cdf4-42cf-8dc7-3df9825dd91d\" (UID: \"b90a7b1a-cdf4-42cf-8dc7-3df9825dd91d\") "
Aug 04 19:06:40.040808 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:06:40.040803    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b90a7b1a-cdf4-42cf-8dc7-3df9825dd91d-cabundle-cert\") pod \"b90a7b1a-cdf4-42cf-8dc7-3df9825dd91d\" (UID: \"b90a7b1a-cdf4-42cf-8dc7-3df9825dd91d\") "
Aug 04 19:06:40.041030 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:06:40.040828    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b90a7b1a-cdf4-42cf-8dc7-3df9825dd91d-model-cache\") pod \"b90a7b1a-cdf4-42cf-8dc7-3df9825dd91d\" (UID: \"b90a7b1a-cdf4-42cf-8dc7-3df9825dd91d\") "
Aug 04 19:06:40.041030 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:06:40.040850    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ztp5g\" (UniqueName: \"kubernetes.io/projected/b90a7b1a-cdf4-42cf-8dc7-3df9825dd91d-kube-api-access-ztp5g\") pod \"b90a7b1a-cdf4-42cf-8dc7-3df9825dd91d\" (UID: \"b90a7b1a-cdf4-42cf-8dc7-3df9825dd91d\") "
Aug 04 19:06:40.041030 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:06:40.040870    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b90a7b1a-cdf4-42cf-8dc7-3df9825dd91d-home\") pod \"b90a7b1a-cdf4-42cf-8dc7-3df9825dd91d\" (UID: \"b90a7b1a-cdf4-42cf-8dc7-3df9825dd91d\") "
Aug 04 19:06:40.041030 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:06:40.040892    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b90a7b1a-cdf4-42cf-8dc7-3df9825dd91d-tmp-dir\") pod \"b90a7b1a-cdf4-42cf-8dc7-3df9825dd91d\" (UID: \"b90a7b1a-cdf4-42cf-8dc7-3df9825dd91d\") "
Aug 04 19:06:40.041030 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:06:40.040920    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b90a7b1a-cdf4-42cf-8dc7-3df9825dd91d-dshm\") pod \"b90a7b1a-cdf4-42cf-8dc7-3df9825dd91d\" (UID: \"b90a7b1a-cdf4-42cf-8dc7-3df9825dd91d\") "
Aug 04 19:06:40.041291 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:06:40.041089    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b90a7b1a-cdf4-42cf-8dc7-3df9825dd91d-model-cache" (OuterVolumeSpecName: "model-cache") pod "b90a7b1a-cdf4-42cf-8dc7-3df9825dd91d" (UID: "b90a7b1a-cdf4-42cf-8dc7-3df9825dd91d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 19:06:40.041291 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:06:40.041215    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b90a7b1a-cdf4-42cf-8dc7-3df9825dd91d-model-cache\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 19:06:40.041291 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:06:40.041280    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b90a7b1a-cdf4-42cf-8dc7-3df9825dd91d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "b90a7b1a-cdf4-42cf-8dc7-3df9825dd91d" (UID: "b90a7b1a-cdf4-42cf-8dc7-3df9825dd91d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 19:06:40.041666 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:06:40.041632    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b90a7b1a-cdf4-42cf-8dc7-3df9825dd91d-home" (OuterVolumeSpecName: "home") pod "b90a7b1a-cdf4-42cf-8dc7-3df9825dd91d" (UID: "b90a7b1a-cdf4-42cf-8dc7-3df9825dd91d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 19:06:40.043080 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:06:40.043045    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b90a7b1a-cdf4-42cf-8dc7-3df9825dd91d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b90a7b1a-cdf4-42cf-8dc7-3df9825dd91d" (UID: "b90a7b1a-cdf4-42cf-8dc7-3df9825dd91d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 19:06:40.043724 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:06:40.043684    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b90a7b1a-cdf4-42cf-8dc7-3df9825dd91d-kube-api-access-ztp5g" (OuterVolumeSpecName: "kube-api-access-ztp5g") pod "b90a7b1a-cdf4-42cf-8dc7-3df9825dd91d" (UID: "b90a7b1a-cdf4-42cf-8dc7-3df9825dd91d"). InnerVolumeSpecName "kube-api-access-ztp5g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 19:06:40.043842 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:06:40.043779    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b90a7b1a-cdf4-42cf-8dc7-3df9825dd91d-dshm" (OuterVolumeSpecName: "dshm") pod "b90a7b1a-cdf4-42cf-8dc7-3df9825dd91d" (UID: "b90a7b1a-cdf4-42cf-8dc7-3df9825dd91d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 19:06:40.053107 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:06:40.053079    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b90a7b1a-cdf4-42cf-8dc7-3df9825dd91d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b90a7b1a-cdf4-42cf-8dc7-3df9825dd91d" (UID: "b90a7b1a-cdf4-42cf-8dc7-3df9825dd91d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 19:06:40.065957 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:06:40.065932    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b90a7b1a-cdf4-42cf-8dc7-3df9825dd91d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b90a7b1a-cdf4-42cf-8dc7-3df9825dd91d" (UID: "b90a7b1a-cdf4-42cf-8dc7-3df9825dd91d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 19:06:40.142374 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:06:40.142346    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ztp5g\" (UniqueName: \"kubernetes.io/projected/b90a7b1a-cdf4-42cf-8dc7-3df9825dd91d-kube-api-access-ztp5g\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 19:06:40.142374 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:06:40.142371    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b90a7b1a-cdf4-42cf-8dc7-3df9825dd91d-home\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 19:06:40.142550 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:06:40.142381    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b90a7b1a-cdf4-42cf-8dc7-3df9825dd91d-tmp-dir\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 19:06:40.142550 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:06:40.142390    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b90a7b1a-cdf4-42cf-8dc7-3df9825dd91d-dshm\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 19:06:40.142550 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:06:40.142398    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b90a7b1a-cdf4-42cf-8dc7-3df9825dd91d-tls-certs\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 19:06:40.142550 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:06:40.142408    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b90a7b1a-cdf4-42cf-8dc7-3df9825dd91d-kserve-provision-location\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 19:06:40.142550 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:06:40.142417    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b90a7b1a-cdf4-42cf-8dc7-3df9825dd91d-cabundle-cert\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 19:06:40.213828 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:06:40.213799    2583 generic.go:356] "Generic (PLEG): container finished" podID="b90a7b1a-cdf4-42cf-8dc7-3df9825dd91d" containerID="bb239d0365fdecd8c6c4fd64067daa6e6e627f6f9f00bef4181f8dc329d3ab98" exitCode=0
Aug 04 19:06:40.214005 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:06:40.213861    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-86ccf66d9fbk9jk" event={"ID":"b90a7b1a-cdf4-42cf-8dc7-3df9825dd91d","Type":"ContainerDied","Data":"bb239d0365fdecd8c6c4fd64067daa6e6e627f6f9f00bef4181f8dc329d3ab98"}
Aug 04 19:06:40.214005 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:06:40.213867    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-86ccf66d9fbk9jk"
Aug 04 19:06:40.214005 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:06:40.213902    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-86ccf66d9fbk9jk" event={"ID":"b90a7b1a-cdf4-42cf-8dc7-3df9825dd91d","Type":"ContainerDied","Data":"830b8c9cffc8bcf2d1981d45057d04bd5c6c9989d8ed697624b3c2021a465bd4"}
Aug 04 19:06:40.214005 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:06:40.213924    2583 scope.go:117] "RemoveContainer" containerID="bb239d0365fdecd8c6c4fd64067daa6e6e627f6f9f00bef4181f8dc329d3ab98"
Aug 04 19:06:40.223316 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:06:40.223294    2583 scope.go:117] "RemoveContainer" containerID="d14ba29b22f896b131dd9adb4bd0decfd43869a55f9ce8c46cd37bc236922565"
Aug 04 19:06:40.231244 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:06:40.231225    2583 scope.go:117] "RemoveContainer" containerID="bb239d0365fdecd8c6c4fd64067daa6e6e627f6f9f00bef4181f8dc329d3ab98"
Aug 04 19:06:40.231534 ip-10-0-135-124 kubenswrapper[2583]: E0804 19:06:40.231509    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb239d0365fdecd8c6c4fd64067daa6e6e627f6f9f00bef4181f8dc329d3ab98\": container with ID starting with bb239d0365fdecd8c6c4fd64067daa6e6e627f6f9f00bef4181f8dc329d3ab98 not found: ID does not exist" containerID="bb239d0365fdecd8c6c4fd64067daa6e6e627f6f9f00bef4181f8dc329d3ab98"
Aug 04 19:06:40.231647 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:06:40.231541    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb239d0365fdecd8c6c4fd64067daa6e6e627f6f9f00bef4181f8dc329d3ab98"} err="failed to get container status \"bb239d0365fdecd8c6c4fd64067daa6e6e627f6f9f00bef4181f8dc329d3ab98\": rpc error: code = NotFound desc = could not find container \"bb239d0365fdecd8c6c4fd64067daa6e6e627f6f9f00bef4181f8dc329d3ab98\": container with ID starting with bb239d0365fdecd8c6c4fd64067daa6e6e627f6f9f00bef4181f8dc329d3ab98 not found: ID does not exist"
Aug 04 19:06:40.231647 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:06:40.231563    2583 scope.go:117] "RemoveContainer" containerID="d14ba29b22f896b131dd9adb4bd0decfd43869a55f9ce8c46cd37bc236922565"
Aug 04 19:06:40.231827 ip-10-0-135-124 kubenswrapper[2583]: E0804 19:06:40.231809    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d14ba29b22f896b131dd9adb4bd0decfd43869a55f9ce8c46cd37bc236922565\": container with ID starting with d14ba29b22f896b131dd9adb4bd0decfd43869a55f9ce8c46cd37bc236922565 not found: ID does not exist" containerID="d14ba29b22f896b131dd9adb4bd0decfd43869a55f9ce8c46cd37bc236922565"
Aug 04 19:06:40.231869 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:06:40.231834    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d14ba29b22f896b131dd9adb4bd0decfd43869a55f9ce8c46cd37bc236922565"} err="failed to get container status \"d14ba29b22f896b131dd9adb4bd0decfd43869a55f9ce8c46cd37bc236922565\": rpc error: code = NotFound desc = could not find container \"d14ba29b22f896b131dd9adb4bd0decfd43869a55f9ce8c46cd37bc236922565\": container with ID starting with d14ba29b22f896b131dd9adb4bd0decfd43869a55f9ce8c46cd37bc236922565 not found: ID does not exist"
Aug 04 19:06:40.238346 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:06:40.238319    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-86ccf66d9fbk9jk"]
Aug 04 19:06:40.241892 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:06:40.241872    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-86ccf66d9fbk9jk"]
Aug 04 19:06:40.856923 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:06:40.856880    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-86ccf66d9fbk9jk" podUID="b90a7b1a-cdf4-42cf-8dc7-3df9825dd91d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Aug 04 19:06:40.942602 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:06:40.942569    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b90a7b1a-cdf4-42cf-8dc7-3df9825dd91d" path="/var/lib/kubelet/pods/b90a7b1a-cdf4-42cf-8dc7-3df9825dd91d/volumes"
Aug 04 19:06:47.625059 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:06:47.625027    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-sdv2m/must-gather-zmxtq"]
Aug 04 19:06:47.625407 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:06:47.625394    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b90a7b1a-cdf4-42cf-8dc7-3df9825dd91d" containerName="main"
Aug 04 19:06:47.625452 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:06:47.625409    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="b90a7b1a-cdf4-42cf-8dc7-3df9825dd91d" containerName="main"
Aug 04 19:06:47.625452 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:06:47.625422    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b90a7b1a-cdf4-42cf-8dc7-3df9825dd91d" containerName="storage-initializer"
Aug 04 19:06:47.625452 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:06:47.625429    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="b90a7b1a-cdf4-42cf-8dc7-3df9825dd91d" containerName="storage-initializer"
Aug 04 19:06:47.625570 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:06:47.625513    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="b90a7b1a-cdf4-42cf-8dc7-3df9825dd91d" containerName="main"
Aug 04 19:06:47.630507 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:06:47.630461    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-sdv2m/must-gather-zmxtq"
Aug 04 19:06:47.633030 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:06:47.633009    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-sdv2m\"/\"default-dockercfg-qp64r\""
Aug 04 19:06:47.634245 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:06:47.634219    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-sdv2m\"/\"kube-root-ca.crt\""
Aug 04 19:06:47.634352 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:06:47.634296    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-sdv2m\"/\"openshift-service-ca.crt\""
Aug 04 19:06:47.637837 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:06:47.637726    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-sdv2m/must-gather-zmxtq"]
Aug 04 19:06:47.703382 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:06:47.703360    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/ccbc9c96-d0ba-4d37-8840-ca80e34e87b8-must-gather-output\") pod \"must-gather-zmxtq\" (UID: \"ccbc9c96-d0ba-4d37-8840-ca80e34e87b8\") " pod="openshift-must-gather-sdv2m/must-gather-zmxtq"
Aug 04 19:06:47.703521 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:06:47.703411    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htrj8\" (UniqueName: \"kubernetes.io/projected/ccbc9c96-d0ba-4d37-8840-ca80e34e87b8-kube-api-access-htrj8\") pod \"must-gather-zmxtq\" (UID: \"ccbc9c96-d0ba-4d37-8840-ca80e34e87b8\") " pod="openshift-must-gather-sdv2m/must-gather-zmxtq"
Aug 04 19:06:47.804359 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:06:47.804329    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htrj8\" (UniqueName: \"kubernetes.io/projected/ccbc9c96-d0ba-4d37-8840-ca80e34e87b8-kube-api-access-htrj8\") pod \"must-gather-zmxtq\" (UID: \"ccbc9c96-d0ba-4d37-8840-ca80e34e87b8\") " pod="openshift-must-gather-sdv2m/must-gather-zmxtq"
Aug 04 19:06:47.804512 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:06:47.804428    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/ccbc9c96-d0ba-4d37-8840-ca80e34e87b8-must-gather-output\") pod \"must-gather-zmxtq\" (UID: \"ccbc9c96-d0ba-4d37-8840-ca80e34e87b8\") " pod="openshift-must-gather-sdv2m/must-gather-zmxtq"
Aug 04 19:06:47.804780 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:06:47.804762    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/ccbc9c96-d0ba-4d37-8840-ca80e34e87b8-must-gather-output\") pod \"must-gather-zmxtq\" (UID: \"ccbc9c96-d0ba-4d37-8840-ca80e34e87b8\") " pod="openshift-must-gather-sdv2m/must-gather-zmxtq"
Aug 04 19:06:47.814233 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:06:47.814205    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-htrj8\" (UniqueName: \"kubernetes.io/projected/ccbc9c96-d0ba-4d37-8840-ca80e34e87b8-kube-api-access-htrj8\") pod \"must-gather-zmxtq\" (UID: \"ccbc9c96-d0ba-4d37-8840-ca80e34e87b8\") " pod="openshift-must-gather-sdv2m/must-gather-zmxtq"
Aug 04 19:06:47.940560 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:06:47.940539    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-sdv2m/must-gather-zmxtq"
Aug 04 19:06:48.078493 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:06:48.078452    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-sdv2m/must-gather-zmxtq"]
Aug 04 19:06:48.080516 ip-10-0-135-124 kubenswrapper[2583]: W0804 19:06:48.080460    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podccbc9c96_d0ba_4d37_8840_ca80e34e87b8.slice/crio-3c5bdb08aff80b0a8740e030e0586e278538b4f8eafac7efed6b07f1327382b0 WatchSource:0}: Error finding container 3c5bdb08aff80b0a8740e030e0586e278538b4f8eafac7efed6b07f1327382b0: Status 404 returned error can't find the container with id 3c5bdb08aff80b0a8740e030e0586e278538b4f8eafac7efed6b07f1327382b0
Aug 04 19:06:48.082313 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:06:48.082297    2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Aug 04 19:06:48.260750 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:06:48.260684    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-sdv2m/must-gather-zmxtq" event={"ID":"ccbc9c96-d0ba-4d37-8840-ca80e34e87b8","Type":"ContainerStarted","Data":"3c5bdb08aff80b0a8740e030e0586e278538b4f8eafac7efed6b07f1327382b0"}
Aug 04 19:06:55.310527 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:06:55.310482    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-sdv2m/must-gather-zmxtq" event={"ID":"ccbc9c96-d0ba-4d37-8840-ca80e34e87b8","Type":"ContainerStarted","Data":"ce9620c2277d007dc256a2060a1e5db803dedc360eb7acc4744463726ec199ea"}
Aug 04 19:06:55.310928 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:06:55.310540    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-sdv2m/must-gather-zmxtq" event={"ID":"ccbc9c96-d0ba-4d37-8840-ca80e34e87b8","Type":"ContainerStarted","Data":"ef7521ad28a0c13688153427f5a96003a624d0e1ebe275d032956a7a32360228"}
Aug 04 19:06:55.330885 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:06:55.330831    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-sdv2m/must-gather-zmxtq" podStartSLOduration=1.980547097 podStartE2EDuration="8.330815419s" totalImagesPullingTime="6.350268322s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 19:06:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:06:55.328125496 +0000 UTC m=+4028.909344086" watchObservedRunningTime="2026-08-04 19:06:55.330815419 +0000 UTC m=+4028.912033996"
Aug 04 19:07:09.824725 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:07:09.824616    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-v894s_ac2a4755-e17f-423b-a44d-0dc532e255f9/discovery/0.log"
Aug 04 19:07:09.893807 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:07:09.893777    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-85c4d8f7fd-bm9rl_88e72b20-e499-46f0-9374-258bf49f39c2/router/0.log"
Aug 04 19:07:10.732686 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:07:10.732653    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-v894s_ac2a4755-e17f-423b-a44d-0dc532e255f9/discovery/0.log"
Aug 04 19:07:10.801917 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:07:10.801883    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-85c4d8f7fd-bm9rl_88e72b20-e499-46f0-9374-258bf49f39c2/router/0.log"
Aug 04 19:07:11.834456 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:07:11.834423    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-m89z5_52e21e58-78e1-499b-a335-8e4991d308b0/manager/0.log"
Aug 04 19:07:11.847844 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:07:11.847819    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-ccbdf686-h9lf5_565c38b3-e6f6-45f9-9c06-c3e84f642f58/manager/0.log"
Aug 04 19:07:11.860301 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:07:11.860279    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-tjxk2_e88b512e-53a6-4cbd-9349-60d49c02f10e/kuadrant-console-plugin/0.log"
Aug 04 19:07:12.061324 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:07:12.061288    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-gr7sm_69c87acd-2bcc-454d-93bc-2ddb9f32d3ea/manager/0.log"
Aug 04 19:07:15.427916 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:07:15.427883    2583 generic.go:356] "Generic (PLEG): container finished" podID="ccbc9c96-d0ba-4d37-8840-ca80e34e87b8" containerID="ef7521ad28a0c13688153427f5a96003a624d0e1ebe275d032956a7a32360228" exitCode=0
Aug 04 19:07:15.428465 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:07:15.427963    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-sdv2m/must-gather-zmxtq" event={"ID":"ccbc9c96-d0ba-4d37-8840-ca80e34e87b8","Type":"ContainerDied","Data":"ef7521ad28a0c13688153427f5a96003a624d0e1ebe275d032956a7a32360228"}
Aug 04 19:07:15.428465 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:07:15.428384    2583 scope.go:117] "RemoveContainer" containerID="ef7521ad28a0c13688153427f5a96003a624d0e1ebe275d032956a7a32360228"
Aug 04 19:07:15.540743 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:07:15.540707    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-sdv2m_must-gather-zmxtq_ccbc9c96-d0ba-4d37-8840-ca80e34e87b8/gather/0.log"
Aug 04 19:07:19.098597 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:07:19.098568    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-lqxxl_31fd88d9-8ce3-4056-a185-4427ec8939c1/global-pull-secret-syncer/0.log"
Aug 04 19:07:19.213359 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:07:19.213327    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-ckl6v_87245716-80f6-43f4-8f06-a0f3924615b2/konnectivity-agent/0.log"
Aug 04 19:07:19.262401 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:07:19.262374    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-135-124.ec2.internal_f727e1c61c596ab926797040f56320d1/haproxy/0.log"
Aug 04 19:07:21.015187 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:07:21.015145    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-sdv2m/must-gather-zmxtq"]
Aug 04 19:07:21.015791 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:07:21.015365    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-must-gather-sdv2m/must-gather-zmxtq" podUID="ccbc9c96-d0ba-4d37-8840-ca80e34e87b8" containerName="copy" containerID="cri-o://ce9620c2277d007dc256a2060a1e5db803dedc360eb7acc4744463726ec199ea" gracePeriod=2
Aug 04 19:07:21.017868 ip-10-0-135-124 kubenswrapper[2583]: E0804 19:07:21.017837    2583 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-zmxtq\" is forbidden: User \"system:node:ip-10-0-135-124.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-sdv2m\": no relationship found between node 'ip-10-0-135-124.ec2.internal' and this object" podUID="ccbc9c96-d0ba-4d37-8840-ca80e34e87b8" pod="openshift-must-gather-sdv2m/must-gather-zmxtq"
Aug 04 19:07:21.018747 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:07:21.018723    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-sdv2m/must-gather-zmxtq"]
Aug 04 19:07:21.253246 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:07:21.253223    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-sdv2m_must-gather-zmxtq_ccbc9c96-d0ba-4d37-8840-ca80e34e87b8/copy/0.log"
Aug 04 19:07:21.254772 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:07:21.254753    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-sdv2m/must-gather-zmxtq"
Aug 04 19:07:21.257165 ip-10-0-135-124 kubenswrapper[2583]: E0804 19:07:21.257141    2583 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-zmxtq\" is forbidden: User \"system:node:ip-10-0-135-124.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-sdv2m\": no relationship found between node 'ip-10-0-135-124.ec2.internal' and this object" podUID="ccbc9c96-d0ba-4d37-8840-ca80e34e87b8" pod="openshift-must-gather-sdv2m/must-gather-zmxtq"
Aug 04 19:07:21.400107 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:07:21.400043    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/ccbc9c96-d0ba-4d37-8840-ca80e34e87b8-must-gather-output\") pod \"ccbc9c96-d0ba-4d37-8840-ca80e34e87b8\" (UID: \"ccbc9c96-d0ba-4d37-8840-ca80e34e87b8\") "
Aug 04 19:07:21.400107 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:07:21.400096    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-htrj8\" (UniqueName: \"kubernetes.io/projected/ccbc9c96-d0ba-4d37-8840-ca80e34e87b8-kube-api-access-htrj8\") pod \"ccbc9c96-d0ba-4d37-8840-ca80e34e87b8\" (UID: \"ccbc9c96-d0ba-4d37-8840-ca80e34e87b8\") "
Aug 04 19:07:21.402229 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:07:21.402202    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccbc9c96-d0ba-4d37-8840-ca80e34e87b8-kube-api-access-htrj8" (OuterVolumeSpecName: "kube-api-access-htrj8") pod "ccbc9c96-d0ba-4d37-8840-ca80e34e87b8" (UID: "ccbc9c96-d0ba-4d37-8840-ca80e34e87b8"). InnerVolumeSpecName "kube-api-access-htrj8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 19:07:21.406989 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:07:21.406965    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccbc9c96-d0ba-4d37-8840-ca80e34e87b8-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "ccbc9c96-d0ba-4d37-8840-ca80e34e87b8" (UID: "ccbc9c96-d0ba-4d37-8840-ca80e34e87b8"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 19:07:21.460650 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:07:21.460627    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-sdv2m_must-gather-zmxtq_ccbc9c96-d0ba-4d37-8840-ca80e34e87b8/copy/0.log"
Aug 04 19:07:21.460968 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:07:21.460947    2583 generic.go:356] "Generic (PLEG): container finished" podID="ccbc9c96-d0ba-4d37-8840-ca80e34e87b8" containerID="ce9620c2277d007dc256a2060a1e5db803dedc360eb7acc4744463726ec199ea" exitCode=143
Aug 04 19:07:21.461019 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:07:21.460991    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-sdv2m/must-gather-zmxtq"
Aug 04 19:07:21.461092 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:07:21.461054    2583 scope.go:117] "RemoveContainer" containerID="ce9620c2277d007dc256a2060a1e5db803dedc360eb7acc4744463726ec199ea"
Aug 04 19:07:21.465632 ip-10-0-135-124 kubenswrapper[2583]: E0804 19:07:21.464049    2583 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-zmxtq\" is forbidden: User \"system:node:ip-10-0-135-124.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-sdv2m\": no relationship found between node 'ip-10-0-135-124.ec2.internal' and this object" podUID="ccbc9c96-d0ba-4d37-8840-ca80e34e87b8" pod="openshift-must-gather-sdv2m/must-gather-zmxtq"
Aug 04 19:07:21.475972 ip-10-0-135-124 kubenswrapper[2583]: E0804 19:07:21.475946    2583 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-zmxtq\" is forbidden: User \"system:node:ip-10-0-135-124.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-sdv2m\": no relationship found between node 'ip-10-0-135-124.ec2.internal' and this object" podUID="ccbc9c96-d0ba-4d37-8840-ca80e34e87b8" pod="openshift-must-gather-sdv2m/must-gather-zmxtq"
Aug 04 19:07:21.476430 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:07:21.476413    2583 scope.go:117] "RemoveContainer" containerID="ef7521ad28a0c13688153427f5a96003a624d0e1ebe275d032956a7a32360228"
Aug 04 19:07:21.488774 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:07:21.488757    2583 scope.go:117] "RemoveContainer" containerID="ce9620c2277d007dc256a2060a1e5db803dedc360eb7acc4744463726ec199ea"
Aug 04 19:07:21.489023 ip-10-0-135-124 kubenswrapper[2583]: E0804 19:07:21.489002    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce9620c2277d007dc256a2060a1e5db803dedc360eb7acc4744463726ec199ea\": container with ID starting with ce9620c2277d007dc256a2060a1e5db803dedc360eb7acc4744463726ec199ea not found: ID does not exist" containerID="ce9620c2277d007dc256a2060a1e5db803dedc360eb7acc4744463726ec199ea"
Aug 04 19:07:21.489075 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:07:21.489031    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce9620c2277d007dc256a2060a1e5db803dedc360eb7acc4744463726ec199ea"} err="failed to get container status \"ce9620c2277d007dc256a2060a1e5db803dedc360eb7acc4744463726ec199ea\": rpc error: code = NotFound desc = could not find container \"ce9620c2277d007dc256a2060a1e5db803dedc360eb7acc4744463726ec199ea\": container with ID starting with ce9620c2277d007dc256a2060a1e5db803dedc360eb7acc4744463726ec199ea not found: ID does not exist"
Aug 04 19:07:21.489075 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:07:21.489049    2583 scope.go:117] "RemoveContainer" containerID="ef7521ad28a0c13688153427f5a96003a624d0e1ebe275d032956a7a32360228"
Aug 04 19:07:21.489280 ip-10-0-135-124 kubenswrapper[2583]: E0804 19:07:21.489262    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef7521ad28a0c13688153427f5a96003a624d0e1ebe275d032956a7a32360228\": container with ID starting with ef7521ad28a0c13688153427f5a96003a624d0e1ebe275d032956a7a32360228 not found: ID does not exist" containerID="ef7521ad28a0c13688153427f5a96003a624d0e1ebe275d032956a7a32360228"
Aug 04 19:07:21.489322 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:07:21.489286    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef7521ad28a0c13688153427f5a96003a624d0e1ebe275d032956a7a32360228"} err="failed to get container status \"ef7521ad28a0c13688153427f5a96003a624d0e1ebe275d032956a7a32360228\": rpc error: code = NotFound desc = could not find container \"ef7521ad28a0c13688153427f5a96003a624d0e1ebe275d032956a7a32360228\": container with ID starting with ef7521ad28a0c13688153427f5a96003a624d0e1ebe275d032956a7a32360228 not found: ID does not exist"
Aug 04 19:07:21.500796 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:07:21.500774    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-htrj8\" (UniqueName: \"kubernetes.io/projected/ccbc9c96-d0ba-4d37-8840-ca80e34e87b8-kube-api-access-htrj8\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 19:07:21.500872 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:07:21.500797    2583 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/ccbc9c96-d0ba-4d37-8840-ca80e34e87b8-must-gather-output\") on node \"ip-10-0-135-124.ec2.internal\" DevicePath \"\""
Aug 04 19:07:22.942997 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:07:22.942959    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ccbc9c96-d0ba-4d37-8840-ca80e34e87b8" path="/var/lib/kubelet/pods/ccbc9c96-d0ba-4d37-8840-ca80e34e87b8/volumes"
Aug 04 19:07:24.073917 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:07:24.073834    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-m89z5_52e21e58-78e1-499b-a335-8e4991d308b0/manager/0.log"
Aug 04 19:07:24.126070 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:07:24.126047    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-ccbdf686-h9lf5_565c38b3-e6f6-45f9-9c06-c3e84f642f58/manager/0.log"
Aug 04 19:07:24.157166 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:07:24.157142    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-tjxk2_e88b512e-53a6-4cbd-9349-60d49c02f10e/kuadrant-console-plugin/0.log"
Aug 04 19:07:24.337868 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:07:24.337748    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-gr7sm_69c87acd-2bcc-454d-93bc-2ddb9f32d3ea/manager/0.log"
Aug 04 19:07:26.090888 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:07:26.090847    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6f55c4f58d-fzk78_e079c95a-25bd-44e2-8f30-44d6cba4add7/cluster-monitoring-operator/0.log"
Aug 04 19:07:26.567105 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:07:26.567073    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9pl9x_50eba630-4be0-4bfe-aa2f-36a5793d8f97/node-exporter/0.log"
Aug 04 19:07:26.596458 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:07:26.596426    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9pl9x_50eba630-4be0-4bfe-aa2f-36a5793d8f97/kube-rbac-proxy/0.log"
Aug 04 19:07:26.629694 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:07:26.629670    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9pl9x_50eba630-4be0-4bfe-aa2f-36a5793d8f97/init-textfile/0.log"
Aug 04 19:07:28.730092 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:07:28.730060    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-f77f48c6d-lt6n7_6d05efcb-b6fb-485c-b472-bd406b8e729f/networking-console-plugin/0.log"
Aug 04 19:07:29.017150 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:07:29.017066    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-m6lzs/perf-node-gather-daemonset-rl4gx"]
Aug 04 19:07:29.017650 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:07:29.017629    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ccbc9c96-d0ba-4d37-8840-ca80e34e87b8" containerName="gather"
Aug 04 19:07:29.017703 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:07:29.017654    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="ccbc9c96-d0ba-4d37-8840-ca80e34e87b8" containerName="gather"
Aug 04 19:07:29.017703 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:07:29.017677    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ccbc9c96-d0ba-4d37-8840-ca80e34e87b8" containerName="copy"
Aug 04 19:07:29.017703 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:07:29.017687    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="ccbc9c96-d0ba-4d37-8840-ca80e34e87b8" containerName="copy"
Aug 04 19:07:29.017837 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:07:29.017781    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="ccbc9c96-d0ba-4d37-8840-ca80e34e87b8" containerName="gather"
Aug 04 19:07:29.017837 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:07:29.017795    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="ccbc9c96-d0ba-4d37-8840-ca80e34e87b8" containerName="copy"
Aug 04 19:07:29.022130 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:07:29.022101    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-m6lzs/perf-node-gather-daemonset-rl4gx"
Aug 04 19:07:29.025041 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:07:29.024977    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-m6lzs\"/\"default-dockercfg-jqf5h\""
Aug 04 19:07:29.025505 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:07:29.025461    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-m6lzs\"/\"openshift-service-ca.crt\""
Aug 04 19:07:29.026395 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:07:29.026361    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-m6lzs\"/\"kube-root-ca.crt\""
Aug 04 19:07:29.029338 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:07:29.029311    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-m6lzs/perf-node-gather-daemonset-rl4gx"]
Aug 04 19:07:29.162257 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:07:29.162216    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/40baca8f-a9a0-4735-b1d7-7ae735bf3b57-lib-modules\") pod \"perf-node-gather-daemonset-rl4gx\" (UID: \"40baca8f-a9a0-4735-b1d7-7ae735bf3b57\") " pod="openshift-must-gather-m6lzs/perf-node-gather-daemonset-rl4gx"
Aug 04 19:07:29.162433 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:07:29.162286    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/40baca8f-a9a0-4735-b1d7-7ae735bf3b57-sys\") pod \"perf-node-gather-daemonset-rl4gx\" (UID: \"40baca8f-a9a0-4735-b1d7-7ae735bf3b57\") " pod="openshift-must-gather-m6lzs/perf-node-gather-daemonset-rl4gx"
Aug 04 19:07:29.162433 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:07:29.162308    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/40baca8f-a9a0-4735-b1d7-7ae735bf3b57-proc\") pod \"perf-node-gather-daemonset-rl4gx\" (UID: \"40baca8f-a9a0-4735-b1d7-7ae735bf3b57\") " pod="openshift-must-gather-m6lzs/perf-node-gather-daemonset-rl4gx"
Aug 04 19:07:29.162433 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:07:29.162324    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/40baca8f-a9a0-4735-b1d7-7ae735bf3b57-podres\") pod \"perf-node-gather-daemonset-rl4gx\" (UID: \"40baca8f-a9a0-4735-b1d7-7ae735bf3b57\") " pod="openshift-must-gather-m6lzs/perf-node-gather-daemonset-rl4gx"
Aug 04 19:07:29.162433 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:07:29.162340    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6bqp\" (UniqueName: \"kubernetes.io/projected/40baca8f-a9a0-4735-b1d7-7ae735bf3b57-kube-api-access-l6bqp\") pod \"perf-node-gather-daemonset-rl4gx\" (UID: \"40baca8f-a9a0-4735-b1d7-7ae735bf3b57\") " pod="openshift-must-gather-m6lzs/perf-node-gather-daemonset-rl4gx"
Aug 04 19:07:29.263266 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:07:29.263226    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/40baca8f-a9a0-4735-b1d7-7ae735bf3b57-sys\") pod \"perf-node-gather-daemonset-rl4gx\" (UID: \"40baca8f-a9a0-4735-b1d7-7ae735bf3b57\") " pod="openshift-must-gather-m6lzs/perf-node-gather-daemonset-rl4gx"
Aug 04 19:07:29.263266 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:07:29.263270    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/40baca8f-a9a0-4735-b1d7-7ae735bf3b57-proc\") pod \"perf-node-gather-daemonset-rl4gx\" (UID: \"40baca8f-a9a0-4735-b1d7-7ae735bf3b57\") " pod="openshift-must-gather-m6lzs/perf-node-gather-daemonset-rl4gx"
Aug 04 19:07:29.263439 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:07:29.263289    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/40baca8f-a9a0-4735-b1d7-7ae735bf3b57-podres\") pod \"perf-node-gather-daemonset-rl4gx\" (UID: \"40baca8f-a9a0-4735-b1d7-7ae735bf3b57\") " pod="openshift-must-gather-m6lzs/perf-node-gather-daemonset-rl4gx"
Aug 04 19:07:29.263439 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:07:29.263307    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6bqp\" (UniqueName: \"kubernetes.io/projected/40baca8f-a9a0-4735-b1d7-7ae735bf3b57-kube-api-access-l6bqp\") pod \"perf-node-gather-daemonset-rl4gx\" (UID: \"40baca8f-a9a0-4735-b1d7-7ae735bf3b57\") " pod="openshift-must-gather-m6lzs/perf-node-gather-daemonset-rl4gx"
Aug 04 19:07:29.263439 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:07:29.263350    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/40baca8f-a9a0-4735-b1d7-7ae735bf3b57-lib-modules\") pod \"perf-node-gather-daemonset-rl4gx\" (UID: \"40baca8f-a9a0-4735-b1d7-7ae735bf3b57\") " pod="openshift-must-gather-m6lzs/perf-node-gather-daemonset-rl4gx"
Aug 04 19:07:29.263439 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:07:29.263362    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/40baca8f-a9a0-4735-b1d7-7ae735bf3b57-sys\") pod \"perf-node-gather-daemonset-rl4gx\" (UID: \"40baca8f-a9a0-4735-b1d7-7ae735bf3b57\") " pod="openshift-must-gather-m6lzs/perf-node-gather-daemonset-rl4gx"
Aug 04 19:07:29.263439 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:07:29.263427    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/40baca8f-a9a0-4735-b1d7-7ae735bf3b57-podres\") pod \"perf-node-gather-daemonset-rl4gx\" (UID: \"40baca8f-a9a0-4735-b1d7-7ae735bf3b57\") " pod="openshift-must-gather-m6lzs/perf-node-gather-daemonset-rl4gx"
Aug 04 19:07:29.263615 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:07:29.263364    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/40baca8f-a9a0-4735-b1d7-7ae735bf3b57-proc\") pod \"perf-node-gather-daemonset-rl4gx\" (UID: \"40baca8f-a9a0-4735-b1d7-7ae735bf3b57\") " pod="openshift-must-gather-m6lzs/perf-node-gather-daemonset-rl4gx"
Aug 04 19:07:29.263615 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:07:29.263487    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/40baca8f-a9a0-4735-b1d7-7ae735bf3b57-lib-modules\") pod \"perf-node-gather-daemonset-rl4gx\" (UID: \"40baca8f-a9a0-4735-b1d7-7ae735bf3b57\") " pod="openshift-must-gather-m6lzs/perf-node-gather-daemonset-rl4gx"
Aug 04 19:07:29.272403 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:07:29.272330    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6bqp\" (UniqueName: \"kubernetes.io/projected/40baca8f-a9a0-4735-b1d7-7ae735bf3b57-kube-api-access-l6bqp\") pod \"perf-node-gather-daemonset-rl4gx\" (UID: \"40baca8f-a9a0-4735-b1d7-7ae735bf3b57\") " pod="openshift-must-gather-m6lzs/perf-node-gather-daemonset-rl4gx"
Aug 04 19:07:29.275499 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:07:29.275445    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-wskfb_6157c2d3-2fd0-4544-88ad-434ac75f8c04/console-operator/1.log"
Aug 04 19:07:29.280129 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:07:29.280101    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-wskfb_6157c2d3-2fd0-4544-88ad-434ac75f8c04/console-operator/2.log"
Aug 04 19:07:29.333524 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:07:29.333451    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-m6lzs/perf-node-gather-daemonset-rl4gx"
Aug 04 19:07:29.463619 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:07:29.463588    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-m6lzs/perf-node-gather-daemonset-rl4gx"]
Aug 04 19:07:29.465763 ip-10-0-135-124 kubenswrapper[2583]: W0804 19:07:29.465724    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod40baca8f_a9a0_4735_b1d7_7ae735bf3b57.slice/crio-0615ac85b3fe2518fc2a653aff2475f44051c13f1bde49d191a8db91fe34ae4b WatchSource:0}: Error finding container 0615ac85b3fe2518fc2a653aff2475f44051c13f1bde49d191a8db91fe34ae4b: Status 404 returned error can't find the container with id 0615ac85b3fe2518fc2a653aff2475f44051c13f1bde49d191a8db91fe34ae4b
Aug 04 19:07:29.504042 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:07:29.504013    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-m6lzs/perf-node-gather-daemonset-rl4gx" event={"ID":"40baca8f-a9a0-4735-b1d7-7ae735bf3b57","Type":"ContainerStarted","Data":"0615ac85b3fe2518fc2a653aff2475f44051c13f1bde49d191a8db91fe34ae4b"}
Aug 04 19:07:29.749119 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:07:29.749085    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-899fffbd7-8t97q_7f355dae-56fa-48cb-a01b-5db54f947f41/console/0.log"
Aug 04 19:07:29.783665 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:07:29.783631    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-77c974ffcf-7whsw_35f07db1-b2dd-45a1-a0d1-eec05bff872f/download-server/0.log"
Aug 04 19:07:30.367760 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:07:30.367724    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-5b8776669-v7q5c_6b6fb1c8-d652-4046-8f8e-4f85375d5d08/volume-data-source-validator/0.log"
Aug 04 19:07:30.510890 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:07:30.510851    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-m6lzs/perf-node-gather-daemonset-rl4gx" event={"ID":"40baca8f-a9a0-4735-b1d7-7ae735bf3b57","Type":"ContainerStarted","Data":"b1b60768a8432d5193bc284aa83503fa18d30e8f7ff64f0b4a934a28322256b6"}
Aug 04 19:07:30.511066 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:07:30.510987    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-m6lzs/perf-node-gather-daemonset-rl4gx"
Aug 04 19:07:30.529139 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:07:30.529093    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-m6lzs/perf-node-gather-daemonset-rl4gx" podStartSLOduration=1.5290800390000001 podStartE2EDuration="1.529080039s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 19:07:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 19:07:30.52767701 +0000 UTC m=+4064.108895587" watchObservedRunningTime="2026-08-04 19:07:30.529080039 +0000 UTC m=+4064.110298615"
Aug 04 19:07:31.339891 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:07:31.339850    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-7wnlf_3fb60b38-9cb9-4a07-9334-1b914ddd4a62/dns/0.log"
Aug 04 19:07:31.362258 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:07:31.362209    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-7wnlf_3fb60b38-9cb9-4a07-9334-1b914ddd4a62/kube-rbac-proxy/0.log"
Aug 04 19:07:31.510907 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:07:31.510875    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-ggrqd_9fffa81f-5a52-4425-b941-1693ba961fba/dns-node-resolver/0.log"
Aug 04 19:07:32.139419 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:07:32.139366    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-nz5rh_00bf4681-2beb-4489-aba4-7e85f7bf2aaa/node-ca/0.log"
Aug 04 19:07:33.286959 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:07:33.286921    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-v894s_ac2a4755-e17f-423b-a44d-0dc532e255f9/discovery/0.log"
Aug 04 19:07:33.381821 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:07:33.381793    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-85c4d8f7fd-bm9rl_88e72b20-e499-46f0-9374-258bf49f39c2/router/0.log"
Aug 04 19:07:33.969808 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:07:33.969781    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-vlvdv_f46d23da-d73f-4ada-8ac2-1a0e3313e664/serve-healthcheck-canary/0.log"
Aug 04 19:07:34.422228 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:07:34.422129    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-77696685b-7mh27_24555f40-5be1-4a70-9e33-ba3c95c1e02f/insights-operator/0.log"
Aug 04 19:07:34.423764 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:07:34.423739    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-77696685b-7mh27_24555f40-5be1-4a70-9e33-ba3c95c1e02f/insights-operator/1.log"
Aug 04 19:07:34.600187 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:07:34.600131    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-xzw5k_31401da2-69b4-4679-8683-63aa4223f38b/kube-rbac-proxy/0.log"
Aug 04 19:07:34.622635 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:07:34.622606    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-xzw5k_31401da2-69b4-4679-8683-63aa4223f38b/exporter/0.log"
Aug 04 19:07:34.644310 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:07:34.644284    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-xzw5k_31401da2-69b4-4679-8683-63aa4223f38b/extractor/0.log"
Aug 04 19:07:36.525627 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:07:36.525597    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-m6lzs/perf-node-gather-daemonset-rl4gx"
Aug 04 19:07:38.453854 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:07:38.453828    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_kserve-controller-manager-5f9d7f69b6-gmpfz_0522cb81-a78a-4bdc-a3c4-aef8c7ac3f24/manager/0.log"
Aug 04 19:07:38.831398 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:07:38.831370    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-v894s_ac2a4755-e17f-423b-a44d-0dc532e255f9/discovery/0.log"
Aug 04 19:07:38.863273 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:07:38.863252    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-7b954ffc95-dwm6d_baa008cd-0d29-44f9-8cf8-aba20b62b76f/server/0.log"
Aug 04 19:07:38.912341 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:07:38.912321    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-85c4d8f7fd-bm9rl_88e72b20-e499-46f0-9374-258bf49f39c2/router/0.log"
Aug 04 19:07:39.790040 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:07:39.790007    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-64568bcd49-p22ln_e5c04e29-0a3f-4a2d-9809-9a62baa34378/seaweedfs/0.log"
Aug 04 19:07:40.152925 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:07:40.152828    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-v894s_ac2a4755-e17f-423b-a44d-0dc532e255f9/discovery/0.log"
Aug 04 19:07:40.230363 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:07:40.230339    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-85c4d8f7fd-bm9rl_88e72b20-e499-46f0-9374-258bf49f39c2/router/0.log"
Aug 04 19:07:44.819243 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:07:44.819206    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7878f4b88c-6m4nc_582f0d25-f53a-491b-ae73-d09ed1ab0501/kube-storage-version-migrator-operator/1.log"
Aug 04 19:07:44.820080 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:07:44.820060    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7878f4b88c-6m4nc_582f0d25-f53a-491b-ae73-d09ed1ab0501/kube-storage-version-migrator-operator/0.log"
Aug 04 19:07:45.818513 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:07:45.818459    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-c6m8x_555d14c0-e9e6-4119-8298-a88258e7c516/kube-multus-additional-cni-plugins/0.log"
Aug 04 19:07:45.845531 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:07:45.845502    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-c6m8x_555d14c0-e9e6-4119-8298-a88258e7c516/egress-router-binary-copy/0.log"
Aug 04 19:07:45.868448 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:07:45.868411    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-c6m8x_555d14c0-e9e6-4119-8298-a88258e7c516/cni-plugins/0.log"
Aug 04 19:07:45.893925 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:07:45.893901    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-c6m8x_555d14c0-e9e6-4119-8298-a88258e7c516/bond-cni-plugin/0.log"
Aug 04 19:07:45.917301 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:07:45.917277    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-c6m8x_555d14c0-e9e6-4119-8298-a88258e7c516/routeoverride-cni/0.log"
Aug 04 19:07:45.939123 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:07:45.939100    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-c6m8x_555d14c0-e9e6-4119-8298-a88258e7c516/whereabouts-cni-bincopy/0.log"
Aug 04 19:07:45.962118 ip-10-0-135-124 kubenswrapper[2583]: I0804 19:07:45.962094    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-c6m8x_555d14c0-e9e6-4119-8298-a88258e7c516/whereabouts-cni/0.log"