Jul 13 18:47:58.091897 ip-10-0-128-215 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Jul 13 18:47:58.091912 ip-10-0-128-215 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Jul 13 18:47:58.091919 ip-10-0-128-215 systemd[1]: kubelet.service: Failed with result 'resources'.
Jul 13 18:47:58.092181 ip-10-0-128-215 systemd[1]: Failed to start Kubernetes Kubelet.
Jul 13 18:48:08.292722 ip-10-0-128-215 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Jul 13 18:48:08.292739 ip-10-0-128-215 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 8ff9ba03e8724509971fd907609321d4 --
Jul 13 18:50:13.356039 ip-10-0-128-215 systemd[1]: Starting Kubernetes Kubelet...
Jul 13 18:50:13.752089 ip-10-0-128-215 kubenswrapper[2565]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 13 18:50:13.752089 ip-10-0-128-215 kubenswrapper[2565]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 13 18:50:13.752089 ip-10-0-128-215 kubenswrapper[2565]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 13 18:50:13.752089 ip-10-0-128-215 kubenswrapper[2565]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 13 18:50:13.752089 ip-10-0-128-215 kubenswrapper[2565]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 13 18:50:13.753294 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.752823    2565 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 13 18:50:13.756454 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.756432    2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 13 18:50:13.756454 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.756453    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 13 18:50:13.756454 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.756457    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 13 18:50:13.756552 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.756460    2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 13 18:50:13.756552 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.756463    2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 13 18:50:13.756552 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.756466    2565 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 13 18:50:13.756552 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.756469    2565 feature_gate.go:328] unrecognized feature gate: Example2
Jul 13 18:50:13.756552 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.756475    2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 13 18:50:13.756552 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.756479    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 13 18:50:13.756552 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.756482    2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 13 18:50:13.756552 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.756484    2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 13 18:50:13.756552 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.756488    2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 13 18:50:13.756552 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.756490    2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 13 18:50:13.756552 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.756492    2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 13 18:50:13.756552 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.756495    2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 13 18:50:13.756552 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.756498    2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 13 18:50:13.756552 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.756500    2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 13 18:50:13.756552 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.756503    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 13 18:50:13.756552 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.756505    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 13 18:50:13.756552 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.756507    2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 13 18:50:13.756552 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.756510    2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 13 18:50:13.756552 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.756512    2565 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 13 18:50:13.756552 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.756517    2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 13 18:50:13.757095 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.756520    2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 13 18:50:13.757095 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.756522    2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 13 18:50:13.757095 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.756525    2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 13 18:50:13.757095 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.756527    2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 13 18:50:13.757095 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.756530    2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 13 18:50:13.757095 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.756532    2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 13 18:50:13.757095 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.756535    2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 13 18:50:13.757095 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.756537    2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 13 18:50:13.757095 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.756540    2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 13 18:50:13.757095 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.756543    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 13 18:50:13.757095 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.756545    2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 13 18:50:13.757095 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.756548    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 13 18:50:13.757095 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.756551    2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 13 18:50:13.757095 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.756554    2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 13 18:50:13.757095 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.756557    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 13 18:50:13.757095 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.756560    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 13 18:50:13.757095 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.756563    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 13 18:50:13.757095 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.756566    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 13 18:50:13.757095 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.756569    2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 13 18:50:13.757549 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.756572    2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 13 18:50:13.757549 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.756574    2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 13 18:50:13.757549 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.756585    2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 13 18:50:13.757549 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.756588    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 13 18:50:13.757549 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.756592    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 13 18:50:13.757549 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.756594    2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 13 18:50:13.757549 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.756597    2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 13 18:50:13.757549 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.756600    2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 13 18:50:13.757549 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.756602    2565 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 13 18:50:13.757549 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.756605    2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 13 18:50:13.757549 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.756607    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 13 18:50:13.757549 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.756610    2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 13 18:50:13.757549 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.756612    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 13 18:50:13.757549 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.756615    2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 13 18:50:13.757549 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.756618    2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 13 18:50:13.757549 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.756620    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 13 18:50:13.757549 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.756624    2565 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 13 18:50:13.757549 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.756627    2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 13 18:50:13.757549 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.756629    2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 13 18:50:13.757549 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.756631    2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 13 18:50:13.758142 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.756634    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 13 18:50:13.758142 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.756636    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 13 18:50:13.758142 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.756639    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 13 18:50:13.758142 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.756641    2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 13 18:50:13.758142 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.756644    2565 feature_gate.go:328] unrecognized feature gate: Example
Jul 13 18:50:13.758142 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.756647    2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 13 18:50:13.758142 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.756650    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 13 18:50:13.758142 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.756653    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 13 18:50:13.758142 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.756655    2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 13 18:50:13.758142 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.756659    2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 13 18:50:13.758142 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.756661    2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 13 18:50:13.758142 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.756665    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 13 18:50:13.758142 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.756668    2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 13 18:50:13.758142 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.756672    2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 13 18:50:13.758142 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.756676    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 13 18:50:13.758142 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.756679    2565 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 13 18:50:13.758142 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.756681    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 13 18:50:13.758142 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.756684    2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 13 18:50:13.758142 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.756686    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 13 18:50:13.758142 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.756689    2565 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 13 18:50:13.758642 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.756692    2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 13 18:50:13.758642 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.756695    2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 13 18:50:13.758642 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.756711    2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 13 18:50:13.758642 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.756714    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 13 18:50:13.758642 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.756717    2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 13 18:50:13.758642 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.756721    2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 13 18:50:13.758642 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.756725    2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 13 18:50:13.758642 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.756728    2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 13 18:50:13.758642 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.756731    2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 13 18:50:13.758642 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.756734    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 13 18:50:13.758642 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.756736    2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 13 18:50:13.758642 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.756740    2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 13 18:50:13.758642 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.756742    2565 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 13 18:50:13.758642 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.756745    2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 13 18:50:13.758642 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.756753    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 13 18:50:13.758642 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.756756    2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 13 18:50:13.758642 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.756759    2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 13 18:50:13.758642 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.757478    2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 13 18:50:13.758642 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.757485    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 13 18:50:13.759126 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.757488    2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 13 18:50:13.759126 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.757490    2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 13 18:50:13.759126 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.757493    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 13 18:50:13.759126 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.757496    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 13 18:50:13.759126 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.757499    2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 13 18:50:13.759126 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.757502    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 13 18:50:13.759126 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.757504    2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 13 18:50:13.759126 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.757507    2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 13 18:50:13.759126 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.757509    2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 13 18:50:13.759126 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.757512    2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 13 18:50:13.759126 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.757514    2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 13 18:50:13.759126 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.757517    2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 13 18:50:13.759126 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.757519    2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 13 18:50:13.759126 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.757522    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 13 18:50:13.759126 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.757525    2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 13 18:50:13.759126 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.757527    2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 13 18:50:13.759126 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.757529    2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 13 18:50:13.759126 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.757532    2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 13 18:50:13.759126 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.757534    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 13 18:50:13.759635 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.757537    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 13 18:50:13.759635 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.757539    2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 13 18:50:13.759635 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.757542    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 13 18:50:13.759635 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.757544    2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 13 18:50:13.759635 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.757547    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 13 18:50:13.759635 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.757550    2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 13 18:50:13.759635 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.757553    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 13 18:50:13.759635 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.757555    2565 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 13 18:50:13.759635 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.757558    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 13 18:50:13.759635 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.757561    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 13 18:50:13.759635 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.757563    2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 13 18:50:13.759635 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.757567    2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 13 18:50:13.759635 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.757569    2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 13 18:50:13.759635 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.757572    2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 13 18:50:13.759635 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.757574    2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 13 18:50:13.759635 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.757577    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 13 18:50:13.759635 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.757580    2565 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 13 18:50:13.759635 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.757583    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 13 18:50:13.759635 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.757585    2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 13 18:50:13.760114 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.757589    2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 13 18:50:13.760114 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.757593    2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 13 18:50:13.760114 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.757596    2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 13 18:50:13.760114 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.757599    2565 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 13 18:50:13.760114 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.757602    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 13 18:50:13.760114 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.757605    2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 13 18:50:13.760114 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.757607    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 13 18:50:13.760114 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.757610    2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 13 18:50:13.760114 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.757613    2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 13 18:50:13.760114 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.757616    2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 13 18:50:13.760114 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.757618    2565 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 13 18:50:13.760114 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.757621    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 13 18:50:13.760114 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.757624    2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 13 18:50:13.760114 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.757627    2565 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 13 18:50:13.760114 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.757630    2565 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 13 18:50:13.760114 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.757632    2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 13 18:50:13.760114 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.757635    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 13 18:50:13.760114 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.757637    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 13 18:50:13.760114 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.757640    2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 13 18:50:13.760114 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.757643    2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 13 18:50:13.760607 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.757645    2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 13 18:50:13.760607 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.757647    2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 13 18:50:13.760607 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.757651    2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 13 18:50:13.760607 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.757655    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 13 18:50:13.760607 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.757658    2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 13 18:50:13.760607 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.757660    2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 13 18:50:13.760607 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.757663    2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 13 18:50:13.760607 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.757665    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 13 18:50:13.760607 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.757668    2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 13 18:50:13.760607 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.757671    2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 13 18:50:13.760607 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.757673    2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 13 18:50:13.760607 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.757676    2565 feature_gate.go:328] unrecognized feature gate: Example
Jul 13 18:50:13.760607 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.757680    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 13 18:50:13.760607 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.757683    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 13 18:50:13.760607 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.757685    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 13 18:50:13.760607 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.757688    2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 13 18:50:13.760607 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.757690    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 13 18:50:13.760607 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.757692    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 13 18:50:13.760607 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.757695    2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 13 18:50:13.760607 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.757716    2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 13 18:50:13.761115 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.757719    2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 13 18:50:13.761115 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.757722    2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 13 18:50:13.761115 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.757724    2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 13 18:50:13.761115 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.757727    2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 13 18:50:13.761115 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.757730    2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 13 18:50:13.761115 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.757732    2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 13 18:50:13.761115 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.757734    2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 13 18:50:13.761115 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.757737    2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 13 18:50:13.761115 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.757740    2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 13 18:50:13.761115 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.757743    2565 feature_gate.go:328] unrecognized feature gate: Example2
Jul 13 18:50:13.761115 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.757745    2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 13 18:50:13.761115 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.757748    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 13 18:50:13.761115 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.757752    2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 13 18:50:13.761115 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.757756    2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 13 18:50:13.761115 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.757759    2565 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 13 18:50:13.761115 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.757762    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 13 18:50:13.761115 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.757765    2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 13 18:50:13.761115 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.757768    2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 13 18:50:13.761115 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.757771    2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 13 18:50:13.761561 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.758242    2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 13 18:50:13.761561 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.758248    2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 13 18:50:13.761561 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.758251    2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 13 18:50:13.761561 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.758254    2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 13 18:50:13.761561 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.758258    2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 13 18:50:13.761561 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.758261    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 13 18:50:13.761561 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.758264    2565 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 13 18:50:13.761561 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.758267    2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 13 18:50:13.761561 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.758270    2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 13 18:50:13.761561 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.758273    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 13 18:50:13.761561 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.758277    2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 13 18:50:13.761561 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.758280    2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 13 18:50:13.761561 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.758283    2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 13 18:50:13.761561 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.758286    2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 13 18:50:13.761561 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.758289    2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 13 18:50:13.761561 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.758291    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 13 18:50:13.761561 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.758294    2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 13 18:50:13.761561 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.758297    2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 13 18:50:13.761561 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.758299    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 13 18:50:13.762035 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.758302    2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 13 18:50:13.762035 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.758306    2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 13 18:50:13.762035 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.758309    2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 13 18:50:13.762035 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.758312    2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 13 18:50:13.762035 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.758315    2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 13 18:50:13.762035 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.758318    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 13 18:50:13.762035 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.758321    2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 13 18:50:13.762035 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.758324    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 13 18:50:13.762035 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.758326    2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 13 18:50:13.762035 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.758329    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 13 18:50:13.762035 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.758333    2565 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 13 18:50:13.762035 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.758336    2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 13 18:50:13.762035 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.758339    2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 13 18:50:13.762035 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.758342    2565 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 13 18:50:13.762035 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.758345    2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 13 18:50:13.762035 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.758347    2565 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 13 18:50:13.762035 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.758350    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 13 18:50:13.762035 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.758352    2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 13 18:50:13.762035 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.758361    2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 13 18:50:13.762035 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.758364    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 13 18:50:13.762503 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.758366    2565 feature_gate.go:328] unrecognized feature gate: Example2
Jul 13 18:50:13.762503 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.758369    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 13 18:50:13.762503 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.758371    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 13 18:50:13.762503 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.758373    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 13 18:50:13.762503 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.758376    2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 13 18:50:13.762503 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.758378    2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 13 18:50:13.762503 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.758381    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 13 18:50:13.762503 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.758384    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 13 18:50:13.762503 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.758386    2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 13 18:50:13.762503 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.758391    2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 13 18:50:13.762503 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.758393    2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 13 18:50:13.762503 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.758396    2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 13 18:50:13.762503 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.758398    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 13 18:50:13.762503 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.758401    2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 13 18:50:13.762503 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.758403    2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 13 18:50:13.762503 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.758405    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 13 18:50:13.762503 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.758408    2565 feature_gate.go:328] unrecognized feature gate: Example
Jul 13 18:50:13.762503 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.758410    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 13 18:50:13.762503 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.758413    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 13 18:50:13.762503 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.758416    2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 13 18:50:13.762991 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.758418    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 13 18:50:13.762991 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.758421    2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 13 18:50:13.762991 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.758423    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 13 18:50:13.762991 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.758426    2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 13 18:50:13.762991 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.758428    2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 13 18:50:13.762991 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.758430    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 13 18:50:13.762991 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.758433    2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 13 18:50:13.762991 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.758435    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 13 18:50:13.762991 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.758438    2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 13 18:50:13.762991 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.758440    2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 13 18:50:13.762991 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.758442    2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 13 18:50:13.762991 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.758451    2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 13 18:50:13.762991 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.758453    2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 13 18:50:13.762991 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.758456    2565 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 13 18:50:13.762991 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.758458    2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 13 18:50:13.762991 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.758460    2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 13 18:50:13.762991 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.758463    2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 13 18:50:13.762991 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.758466    2565 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 13 18:50:13.762991 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.758468    2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 13 18:50:13.762991 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.758471    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 13 18:50:13.763464 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.758473    2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 13 18:50:13.763464 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.758476    2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 13 18:50:13.763464 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.758479    2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 13 18:50:13.763464 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.758482    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 13 18:50:13.763464 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.758485    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 13 18:50:13.763464 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.758487    2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 13 18:50:13.763464 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.758489    2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 13 18:50:13.763464 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.758492    2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 13 18:50:13.763464 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.758494    2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 13 18:50:13.763464 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.758497    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 13 18:50:13.763464 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.758499    2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 13 18:50:13.763464 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.758502    2565 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 13 18:50:13.763464 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.758504    2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 13 18:50:13.763464 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.758508    2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 13 18:50:13.763464 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.758512    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 13 18:50:13.763464 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.758515    2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 13 18:50:13.763464 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.758517    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 13 18:50:13.763464 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.758520    2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 13 18:50:13.763464 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.758522    2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 13 18:50:13.763934 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.758525    2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 13 18:50:13.763934 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.758603    2565 flags.go:64] FLAG: --address="0.0.0.0"
Jul 13 18:50:13.763934 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.758611    2565 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 13 18:50:13.763934 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.758623    2565 flags.go:64] FLAG: --anonymous-auth="true"
Jul 13 18:50:13.763934 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.758628    2565 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 13 18:50:13.763934 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.758642    2565 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 13 18:50:13.763934 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.758646    2565 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 13 18:50:13.763934 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.758651    2565 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 13 18:50:13.763934 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.758655    2565 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 13 18:50:13.763934 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.758659    2565 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 13 18:50:13.763934 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.758661    2565 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 13 18:50:13.763934 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.758666    2565 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 13 18:50:13.763934 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.758669    2565 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 13 18:50:13.763934 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.758672    2565 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 13 18:50:13.763934 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.758676    2565 flags.go:64] FLAG: --cgroup-root=""
Jul 13 18:50:13.763934 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.758679    2565 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 13 18:50:13.763934 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.758682    2565 flags.go:64] FLAG: --client-ca-file=""
Jul 13 18:50:13.763934 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.758685    2565 flags.go:64] FLAG: --cloud-provider="external"
Jul 13 18:50:13.763934 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.758688    2565 flags.go:64] FLAG: --cluster-dns="[]"
Jul 13 18:50:13.763934 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.758693    2565 flags.go:64] FLAG: --cluster-domain=""
Jul 13 18:50:13.763934 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.758696    2565 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 13 18:50:13.763934 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.758714    2565 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 13 18:50:13.763934 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.758718    2565 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 13 18:50:13.763934 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.758721    2565 flags.go:64] FLAG: --container-log-max-files="5"
Jul 13 18:50:13.764503 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.758725    2565 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 13 18:50:13.764503 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.758728    2565 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 13 18:50:13.764503 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.758731    2565 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 13 18:50:13.764503 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.758734    2565 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 13 18:50:13.764503 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.758737    2565 flags.go:64] FLAG: --contention-profiling="false"
Jul 13 18:50:13.764503 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.758740    2565 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 13 18:50:13.764503 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.758743    2565 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 13 18:50:13.764503 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.758746    2565 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 13 18:50:13.764503 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.758749    2565 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 13 18:50:13.764503 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.758753    2565 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 13 18:50:13.764503 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.758756    2565 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 13 18:50:13.764503 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.758759    2565 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 13 18:50:13.764503 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.758761    2565 flags.go:64] FLAG: --enable-load-reader="false"
Jul 13 18:50:13.764503 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.758764    2565 flags.go:64] FLAG: --enable-server="true"
Jul 13 18:50:13.764503 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.758768    2565 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 13 18:50:13.764503 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.758772    2565 flags.go:64] FLAG: --event-burst="100"
Jul 13 18:50:13.764503 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.758776    2565 flags.go:64] FLAG: --event-qps="50"
Jul 13 18:50:13.764503 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.758780    2565 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 13 18:50:13.764503 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.758785    2565 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 13 18:50:13.764503 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.758789    2565 flags.go:64] FLAG: --eviction-hard=""
Jul 13 18:50:13.764503 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.758795    2565 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 13 18:50:13.764503 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.758799    2565 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 13 18:50:13.764503 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.758804    2565 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 13 18:50:13.764503 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.758808    2565 flags.go:64] FLAG: --eviction-soft=""
Jul 13 18:50:13.764503 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.758810    2565 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 13 18:50:13.765182 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.758813    2565 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 13 18:50:13.765182 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.758816    2565 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 13 18:50:13.765182 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.758819    2565 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 13 18:50:13.765182 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.758821    2565 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 13 18:50:13.765182 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.758824    2565 flags.go:64] FLAG: --fail-swap-on="true"
Jul 13 18:50:13.765182 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.758827    2565 flags.go:64] FLAG: --feature-gates=""
Jul 13 18:50:13.765182 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.758832    2565 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 13 18:50:13.765182 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.758835    2565 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 13 18:50:13.765182 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.758838    2565 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 13 18:50:13.765182 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.758842    2565 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 13 18:50:13.765182 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.758845    2565 flags.go:64] FLAG: --healthz-port="10248"
Jul 13 18:50:13.765182 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.758848    2565 flags.go:64] FLAG: --help="false"
Jul 13 18:50:13.765182 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.758851    2565 flags.go:64] FLAG: --hostname-override="ip-10-0-128-215.ec2.internal"
Jul 13 18:50:13.765182 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.758854    2565 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 13 18:50:13.765182 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.758857    2565 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 13 18:50:13.765182 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.758860    2565 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 13 18:50:13.765182 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.758864    2565 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 13 18:50:13.765182 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.758867    2565 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 13 18:50:13.765182 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.758870    2565 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 13 18:50:13.765182 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.758873    2565 flags.go:64] FLAG: --image-service-endpoint=""
Jul 13 18:50:13.765182 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.758876    2565 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 13 18:50:13.765182 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.758879    2565 flags.go:64] FLAG: --kube-api-burst="100"
Jul 13 18:50:13.765182 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.758882    2565 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 13 18:50:13.765835 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.758886    2565 flags.go:64] FLAG: --kube-api-qps="50"
Jul 13 18:50:13.765835 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.758888    2565 flags.go:64] FLAG: --kube-reserved=""
Jul 13 18:50:13.765835 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.758891    2565 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 13 18:50:13.765835 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.758894    2565 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 13 18:50:13.765835 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.758897    2565 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 13 18:50:13.765835 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.758900    2565 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 13 18:50:13.765835 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.758904    2565 flags.go:64] FLAG: --lock-file=""
Jul 13 18:50:13.765835 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.758907    2565 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 13 18:50:13.765835 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.758910    2565 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 13 18:50:13.765835 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.758913    2565 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 13 18:50:13.765835 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.758919    2565 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 13 18:50:13.765835 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.758922    2565 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 13 18:50:13.765835 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.758925    2565 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 13 18:50:13.765835 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.758928    2565 flags.go:64] FLAG: --logging-format="text"
Jul 13 18:50:13.765835 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.758931    2565 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 13 18:50:13.765835 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.758936    2565 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 13 18:50:13.765835 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.758939    2565 flags.go:64] FLAG: --manifest-url=""
Jul 13 18:50:13.765835 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.758942    2565 flags.go:64] FLAG: --manifest-url-header=""
Jul 13 18:50:13.765835 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.758947    2565 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 13 18:50:13.765835 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.758950    2565 flags.go:64] FLAG: --max-open-files="1000000"
Jul 13 18:50:13.765835 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.758954    2565 flags.go:64] FLAG: --max-pods="110"
Jul 13 18:50:13.765835 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.758957    2565 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 13 18:50:13.765835 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.758961    2565 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 13 18:50:13.765835 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.758963    2565 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 13 18:50:13.765835 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.758966    2565 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 13 18:50:13.766449 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.758969    2565 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 13 18:50:13.766449 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.758972    2565 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 13 18:50:13.766449 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.758975    2565 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 13 18:50:13.766449 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.758983    2565 flags.go:64] FLAG: --node-status-max-images="50"
Jul 13 18:50:13.766449 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.758986    2565 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 13 18:50:13.766449 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.758989    2565 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 13 18:50:13.766449 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.758992    2565 flags.go:64] FLAG: --pod-cidr=""
Jul 13 18:50:13.766449 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.758996    2565 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b04c0b1f73af4d021dd8f0b51bfeb302c24482c093de5f2c43654ab56c506926"
Jul 13 18:50:13.766449 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.759002    2565 flags.go:64] FLAG: --pod-manifest-path=""
Jul 13 18:50:13.766449 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.759005    2565 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 13 18:50:13.766449 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.759008    2565 flags.go:64] FLAG: --pods-per-core="0"
Jul 13 18:50:13.766449 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.759011    2565 flags.go:64] FLAG: --port="10250"
Jul 13 18:50:13.766449 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.759015    2565 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 13 18:50:13.766449 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.759019    2565 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0500e124c83c6f265"
Jul 13 18:50:13.766449 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.759023    2565 flags.go:64] FLAG: --qos-reserved=""
Jul 13 18:50:13.766449 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.759026    2565 flags.go:64] FLAG: --read-only-port="10255"
Jul 13 18:50:13.766449 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.759029    2565 flags.go:64] FLAG: --register-node="true"
Jul 13 18:50:13.766449 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.759032    2565 flags.go:64] FLAG: --register-with-taints=""
Jul 13 18:50:13.766449 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.759035    2565 flags.go:64] FLAG: --registry-burst="10"
Jul 13 18:50:13.766449 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.759038    2565 flags.go:64] FLAG: --registry-qps="5"
Jul 13 18:50:13.766449 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.759041    2565 flags.go:64] FLAG: --reserved-cpus=""
Jul 13 18:50:13.766449 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.759043    2565 flags.go:64] FLAG: --reserved-memory=""
Jul 13 18:50:13.766449 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.759047    2565 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 13 18:50:13.766449 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.759052    2565 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 13 18:50:13.766449 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.759055    2565 flags.go:64] FLAG: --rotate-certificates="false"
Jul 13 18:50:13.767075 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.759057    2565 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 13 18:50:13.767075 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.759060    2565 flags.go:64] FLAG: --runonce="false"
Jul 13 18:50:13.767075 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.759063    2565 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 13 18:50:13.767075 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.759066    2565 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 13 18:50:13.767075 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.759069    2565 flags.go:64] FLAG: --seccomp-default="false"
Jul 13 18:50:13.767075 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.759072    2565 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 13 18:50:13.767075 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.759074    2565 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 13 18:50:13.767075 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.759077    2565 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 13 18:50:13.767075 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.759080    2565 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 13 18:50:13.767075 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.759084    2565 flags.go:64] FLAG: --storage-driver-password="root"
Jul 13 18:50:13.767075 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.759086    2565 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 13 18:50:13.767075 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.759089    2565 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 13 18:50:13.767075 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.759092    2565 flags.go:64] FLAG: --storage-driver-user="root"
Jul 13 18:50:13.767075 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.759095    2565 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 13 18:50:13.767075 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.759098    2565 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 13 18:50:13.767075 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.759102    2565 flags.go:64] FLAG: --system-cgroups=""
Jul 13 18:50:13.767075 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.759104    2565 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 13 18:50:13.767075 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.759110    2565 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 13 18:50:13.767075 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.759112    2565 flags.go:64] FLAG: --tls-cert-file=""
Jul 13 18:50:13.767075 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.759115    2565 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 13 18:50:13.767075 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.759128    2565 flags.go:64] FLAG: --tls-min-version=""
Jul 13 18:50:13.767075 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.759132    2565 flags.go:64] FLAG: --tls-private-key-file=""
Jul 13 18:50:13.767075 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.759135    2565 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 13 18:50:13.767075 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.759138    2565 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 13 18:50:13.767075 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.759141    2565 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 13 18:50:13.767679 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.759144    2565 flags.go:64] FLAG: --v="2"
Jul 13 18:50:13.767679 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.759152    2565 flags.go:64] FLAG: --version="false"
Jul 13 18:50:13.767679 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.759156    2565 flags.go:64] FLAG: --vmodule=""
Jul 13 18:50:13.767679 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.759160    2565 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 13 18:50:13.767679 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.759164    2565 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 13 18:50:13.767679 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.759290    2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 13 18:50:13.767679 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.759294    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 13 18:50:13.767679 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.759297    2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 13 18:50:13.767679 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.759300    2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 13 18:50:13.767679 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.759303    2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 13 18:50:13.767679 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.759306    2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 13 18:50:13.767679 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.759309    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 13 18:50:13.767679 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.759312    2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 13 18:50:13.767679 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.759315    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 13 18:50:13.767679 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.759318    2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 13 18:50:13.767679 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.759321    2565 feature_gate.go:328] unrecognized feature gate: Example
Jul 13 18:50:13.767679 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.759323    2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 13 18:50:13.767679 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.759326    2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 13 18:50:13.767679 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.759328    2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 13 18:50:13.767679 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.759331    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 13 18:50:13.767679 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.759333    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 13 18:50:13.768265 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.759336    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 13 18:50:13.768265 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.759340    2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 13 18:50:13.768265 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.759344    2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 13 18:50:13.768265 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.759347    2565 feature_gate.go:328] unrecognized feature gate: Example2
Jul 13 18:50:13.768265 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.759350    2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 13 18:50:13.768265 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.759353    2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 13 18:50:13.768265 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.759357    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 13 18:50:13.768265 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.759360    2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 13 18:50:13.768265 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.759362    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 13 18:50:13.768265 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.759365    2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 13 18:50:13.768265 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.759367    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 13 18:50:13.768265 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.759370    2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 13 18:50:13.768265 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.759373    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 13 18:50:13.768265 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.759375    2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 13 18:50:13.768265 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.759378    2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 13 18:50:13.768265 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.759380    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 13 18:50:13.768265 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.759384    2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 13 18:50:13.768265 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.759386    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 13 18:50:13.768265 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.759389    2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 13 18:50:13.768797 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.759391    2565 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 13 18:50:13.768797 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.759395    2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 13 18:50:13.768797 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.759398    2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 13 18:50:13.768797 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.759401    2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 13 18:50:13.768797 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.759404    2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 13 18:50:13.768797 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.759407    2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 13 18:50:13.768797 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.759410    2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 13 18:50:13.768797 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.759413    2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 13 18:50:13.768797 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.759415    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 13 18:50:13.768797 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.759418    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 13 18:50:13.768797 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.759420    2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 13 18:50:13.768797 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.759423    2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 13 18:50:13.768797 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.759425    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 13 18:50:13.768797 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.759428    2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 13 18:50:13.768797 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.759430    2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 13 18:50:13.768797 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.759433    2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 13 18:50:13.768797 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.759436    2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 13 18:50:13.768797 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.759439    2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 13 18:50:13.768797 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.759441    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 13 18:50:13.768797 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.759445    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 13 18:50:13.769286 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.759447    2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 13 18:50:13.769286 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.759450    2565 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 13 18:50:13.769286 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.759453    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 13 18:50:13.769286 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.759455    2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 13 18:50:13.769286 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.759458    2565 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 13 18:50:13.769286 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.759460    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 13 18:50:13.769286 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.759463    2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 13 18:50:13.769286 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.759465    2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 13 18:50:13.769286 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.759468    2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 13 18:50:13.769286 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.759471    2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 13 18:50:13.769286 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.759474    2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 13 18:50:13.769286 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.759476    2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 13 18:50:13.769286 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.759479    2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 13 18:50:13.769286 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.759482    2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 13 18:50:13.769286 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.759484    2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 13 18:50:13.769286 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.759486    2565 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 13 18:50:13.769286 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.759489    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 13 18:50:13.769286 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.759493    2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 13 18:50:13.769286 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.759496    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 13 18:50:13.769751 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.759499    2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 13 18:50:13.769751 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.759502    2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 13 18:50:13.769751 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.759504    2565 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 13 18:50:13.769751 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.759507    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 13 18:50:13.769751 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.759509    2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 13 18:50:13.769751 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.759512    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 13 18:50:13.769751 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.759514    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 13 18:50:13.769751 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.759517    2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 13 18:50:13.769751 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.759519    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 13 18:50:13.769751 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.759522    2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 13 18:50:13.769751 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.759525    2565 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 13 18:50:13.769751 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.759527    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 13 18:50:13.769751 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.759531    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 13 18:50:13.769751 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.759534    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 13 18:50:13.769751 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.759536    2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 13 18:50:13.769751 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.759539    2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 13 18:50:13.769751 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.759541    2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 13 18:50:13.769751 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.759543    2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 13 18:50:13.769751 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.759546    2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 13 18:50:13.769751 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.759548    2565 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 13 18:50:13.770231 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.759551    2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 13 18:50:13.770231 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.759554    2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 13 18:50:13.770231 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.759557    2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 13 18:50:13.770231 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.759560    2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 13 18:50:13.770231 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.759562    2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 13 18:50:13.770231 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.760118    2565 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 13 18:50:13.770231 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.767453    2565 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 13 18:50:13.770231 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.767476    2565 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 13 18:50:13.770231 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.767492    2565 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 13 18:50:13.770231 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.767498    2565 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 13 18:50:13.770231 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.767568    2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 13 18:50:13.770231 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.767574    2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 13 18:50:13.770231 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.767577    2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 13 18:50:13.770231 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.767580    2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 13 18:50:13.770231 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.767583    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 13 18:50:13.770599 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.767585    2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 13 18:50:13.770599 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.767588    2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 13 18:50:13.770599 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.767591    2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 13 18:50:13.770599 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.767593    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 13 18:50:13.770599 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.767596    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 13 18:50:13.770599 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.767599    2565 feature_gate.go:328] unrecognized feature gate: Example2
Jul 13 18:50:13.770599 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.767601    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 13 18:50:13.770599 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.767603    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 13 18:50:13.770599 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.767606    2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 13 18:50:13.770599 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.767608    2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 13 18:50:13.770599 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.767611    2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 13 18:50:13.770599 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.767614    2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 13 18:50:13.770599 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.767616    2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 13 18:50:13.770599 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.767619    2565 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 13 18:50:13.770599 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.767621    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 13 18:50:13.770599 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.767624    2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 13 18:50:13.770599 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.767626    2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 13 18:50:13.770599 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.767629    2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 13 18:50:13.770599 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.767631    2565 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 13 18:50:13.770599 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.767633    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 13 18:50:13.771151 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.767636    2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 13 18:50:13.771151 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.767638    2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 13 18:50:13.771151 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.767641    2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 13 18:50:13.771151 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.767643    2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 13 18:50:13.771151 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.767655    2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 13 18:50:13.771151 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.767658    2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 13 18:50:13.771151 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.767660    2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 13 18:50:13.771151 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.767663    2565 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 13 18:50:13.771151 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.767666    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 13 18:50:13.771151 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.767668    2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 13 18:50:13.771151 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.767671    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 13 18:50:13.771151 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.767673    2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 13 18:50:13.771151 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.767678    2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 13 18:50:13.771151 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.767683    2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 13 18:50:13.771151 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.767687    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 13 18:50:13.771151 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.767689    2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 13 18:50:13.771151 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.767692    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 13 18:50:13.771151 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.767695    2565 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 13 18:50:13.771151 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.767715    2565 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 13 18:50:13.771625 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.767719    2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 13 18:50:13.771625 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.767722    2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 13 18:50:13.771625 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.767725    2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 13 18:50:13.771625 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.767728    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 13 18:50:13.771625 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.767730    2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 13 18:50:13.771625 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.767733    2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 13 18:50:13.771625 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.767735    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 13 18:50:13.771625 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.767738    2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 13 18:50:13.771625 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.767740    2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 13 18:50:13.771625 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.767742    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 13 18:50:13.771625 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.767745    2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 13 18:50:13.771625 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.767748    2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 13 18:50:13.771625 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.767750    2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 13 18:50:13.771625 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.767752    2565 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 13 18:50:13.771625 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.767755    2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 13 18:50:13.771625 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.767758    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 13 18:50:13.771625 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.767760    2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 13 18:50:13.771625 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.767763    2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 13 18:50:13.771625 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.767773    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 13 18:50:13.771625 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.767776    2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 13 18:50:13.772234 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.767778    2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 13 18:50:13.772234 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.767781    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 13 18:50:13.772234 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.767784    2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 13 18:50:13.772234 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.767788    2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 13 18:50:13.772234 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.767791    2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 13 18:50:13.772234 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.767794    2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 13 18:50:13.772234 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.767797    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 13 18:50:13.772234 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.767799    2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 13 18:50:13.772234 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.767802    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 13 18:50:13.772234 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.767805    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 13 18:50:13.772234 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.767808    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 13 18:50:13.772234 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.767810    2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 13 18:50:13.772234 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.767813    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 13 18:50:13.772234 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.767816    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 13 18:50:13.772234 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.767818    2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 13 18:50:13.772234 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.767821    2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 13 18:50:13.772234 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.767823    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 13 18:50:13.772234 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.767825    2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 13 18:50:13.772234 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.767829    2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 13 18:50:13.772843 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.767833    2565 feature_gate.go:328] unrecognized feature gate: Example
Jul 13 18:50:13.772843 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.767835    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 13 18:50:13.772843 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.767838    2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 13 18:50:13.772843 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.767840    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 13 18:50:13.772843 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.767843    2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 13 18:50:13.772843 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.767845    2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 13 18:50:13.772843 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.767848    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 13 18:50:13.772843 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.767850    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 13 18:50:13.772843 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.767853    2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 13 18:50:13.772843 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.767855    2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 13 18:50:13.772843 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.767859    2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 13 18:50:13.772843 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.767861    2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 13 18:50:13.772843 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.767869    2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 13 18:50:13.772843 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.767872    2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 13 18:50:13.772843 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.767875    2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 13 18:50:13.772843 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.767877    2565 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 13 18:50:13.773253 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.767882    2565 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 13 18:50:13.773253 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.768039    2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 13 18:50:13.773253 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.768045    2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 13 18:50:13.773253 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.768047    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 13 18:50:13.773253 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.768050    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 13 18:50:13.773253 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.768053    2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 13 18:50:13.773253 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.768055    2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 13 18:50:13.773253 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.768059    2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 13 18:50:13.773253 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.768063    2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 13 18:50:13.773253 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.768066    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 13 18:50:13.773253 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.768069    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 13 18:50:13.773253 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.768071    2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 13 18:50:13.773253 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.768074    2565 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 13 18:50:13.773253 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.768076    2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 13 18:50:13.773634 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.768079    2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 13 18:50:13.773634 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.768081    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 13 18:50:13.773634 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.768084    2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 13 18:50:13.773634 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.768087    2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 13 18:50:13.773634 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.768091    2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 13 18:50:13.773634 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.768093    2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 13 18:50:13.773634 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.768096    2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 13 18:50:13.773634 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.768099    2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 13 18:50:13.773634 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.768102    2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 13 18:50:13.773634 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.768105    2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 13 18:50:13.773634 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.768107    2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 13 18:50:13.773634 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.768110    2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 13 18:50:13.773634 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.768113    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 13 18:50:13.773634 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.768116    2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 13 18:50:13.773634 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.768119    2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 13 18:50:13.773634 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.768128    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 13 18:50:13.773634 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.768131    2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 13 18:50:13.773634 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.768133    2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 13 18:50:13.773634 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.768136    2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 13 18:50:13.774115 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.768139    2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 13 18:50:13.774115 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.768141    2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 13 18:50:13.774115 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.768143    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 13 18:50:13.774115 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.768146    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 13 18:50:13.774115 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.768148    2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 13 18:50:13.774115 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.768151    2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 13 18:50:13.774115 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.768154    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 13 18:50:13.774115 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.768157    2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 13 18:50:13.774115 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.768161    2565 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 13 18:50:13.774115 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.768163    2565 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 13 18:50:13.774115 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.768166    2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 13 18:50:13.774115 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.768169    2565 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 13 18:50:13.774115 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.768173    2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 13 18:50:13.774115 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.768175    2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 13 18:50:13.774115 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.768178    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 13 18:50:13.774115 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.768180    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 13 18:50:13.774115 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.768183    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 13 18:50:13.774115 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.768185    2565 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 13 18:50:13.774115 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.768188    2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 13 18:50:13.774115 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.768190    2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 13 18:50:13.774739 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.768193    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 13 18:50:13.774739 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.768195    2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 13 18:50:13.774739 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.768197    2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 13 18:50:13.774739 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.768200    2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 13 18:50:13.774739 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.768202    2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 13 18:50:13.774739 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.768205    2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 13 18:50:13.774739 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.768208    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 13 18:50:13.774739 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.768210    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 13 18:50:13.774739 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.768213    2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 13 18:50:13.774739 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.768221    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 13 18:50:13.774739 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.768224    2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 13 18:50:13.774739 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.768226    2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 13 18:50:13.774739 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.768229    2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 13 18:50:13.774739 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.768231    2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 13 18:50:13.774739 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.768234    2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 13 18:50:13.774739 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.768236    2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 13 18:50:13.774739 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.768238    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 13 18:50:13.774739 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.768241    2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 13 18:50:13.774739 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.768243    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 13 18:50:13.774739 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.768246    2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 13 18:50:13.775382 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.768248    2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 13 18:50:13.775382 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.768250    2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 13 18:50:13.775382 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.768253    2565 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 13 18:50:13.775382 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.768255    2565 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 13 18:50:13.775382 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.768258    2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 13 18:50:13.775382 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.768260    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 13 18:50:13.775382 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.768262    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 13 18:50:13.775382 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.768265    2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 13 18:50:13.775382 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.768267    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 13 18:50:13.775382 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.768270    2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 13 18:50:13.775382 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.768272    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 13 18:50:13.775382 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.768275    2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 13 18:50:13.775382 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.768278    2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 13 18:50:13.775382 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.768281    2565 feature_gate.go:328] unrecognized feature gate: Example2
Jul 13 18:50:13.775382 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.768284    2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 13 18:50:13.775382 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.768286    2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 13 18:50:13.775382 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.768289    2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 13 18:50:13.775382 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.768292    2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 13 18:50:13.775382 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.768294    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 13 18:50:13.775382 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.768297    2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 13 18:50:13.776190 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.768299    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 13 18:50:13.776190 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.768302    2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 13 18:50:13.776190 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.768310    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 13 18:50:13.776190 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.768313    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 13 18:50:13.776190 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.768315    2565 feature_gate.go:328] unrecognized feature gate: Example
Jul 13 18:50:13.776190 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.768318    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 13 18:50:13.776190 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:13.768320    2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 13 18:50:13.776190 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.768326    2565 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 13 18:50:13.776190 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.768506    2565 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 13 18:50:13.776190 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.770584    2565 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 13 18:50:13.776190 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.772043    2565 server.go:1019] "Starting client certificate rotation"
Jul 13 18:50:13.776190 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.772211    2565 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 13 18:50:13.776190 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.772329    2565 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 13 18:50:13.776640 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.774757    2565 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 13 18:50:13.781148 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.781126    2565 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 13 18:50:13.790031 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.790010    2565 log.go:25] "Validated CRI v1 runtime API"
Jul 13 18:50:13.792171 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.792153    2565 log.go:25] "Validated CRI v1 image API"
Jul 13 18:50:13.792970 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.792954    2565 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 13 18:50:13.796489 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.796459    2565 fs.go:135] Filesystem UUIDs: map[0fe08cbb-ff7b-48a7-8e47-f0a363353758:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 f420b9e9-6170-48e8-9471-b1d79b1efeab:/dev/nvme0n1p4]
Jul 13 18:50:13.796544 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.796488    2565 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 13 18:50:13.802130 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.802014    2565 manager.go:216] Machine: {Timestamp:2026-07-13 18:50:13.800726851 +0000 UTC m=+0.325860275 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3106082 MemoryCapacity:32812175360 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec26de3e2d24195875c55c5bdf12a375 SystemUUID:ec26de3e-2d24-1958-75c5-5c5bdf12a375 BootID:8ff9ba03-e872-4509-971f-d907609321d4 Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406089728 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406085632 Type:vfs Inodes:4005392 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ad:46:e9:8b:65 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ad:46:e9:8b:65 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ca:77:5b:28:5d:49 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812175360 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 13 18:50:13.802130 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.802125    2565 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 13 18:50:13.802259 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.802223    2565 manager.go:232] Version: {KernelVersion:5.14.0-570.126.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260702-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 13 18:50:13.803236 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.803221    2565 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 13 18:50:13.805040 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.805009    2565 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 13 18:50:13.805200 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.805045    2565 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-128-215.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 13 18:50:13.805756 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.805745    2565 topology_manager.go:138] "Creating topology manager with none policy"
Jul 13 18:50:13.805790 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.805758    2565 container_manager_linux.go:309] "Creating device plugin manager"
Jul 13 18:50:13.805790 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.805766    2565 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 13 18:50:13.806445 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.806434    2565 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 13 18:50:13.807208 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.807198    2565 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 13 18:50:13.809891 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.809876    2565 state_mem.go:36] "Initialized new in-memory state store"
Jul 13 18:50:13.810042 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.810032    2565 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 13 18:50:13.810076 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.810070    2565 kubelet.go:486] "Attempting to sync node with API server"
Jul 13 18:50:13.810106 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.810100    2565 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 13 18:50:13.810134 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.810120    2565 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 13 18:50:13.810163 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.810135    2565 kubelet.go:398] "Adding apiserver pod source"
Jul 13 18:50:13.810163 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.810148    2565 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 13 18:50:13.810715 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.810691    2565 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 13 18:50:13.810789 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.810729    2565 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 13 18:50:13.813302 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.813277    2565 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 13 18:50:13.815049 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.815033    2565 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 13 18:50:13.815133 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.815064    2565 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 13 18:50:13.815746 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.815730    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 13 18:50:13.815816 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.815751    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 13 18:50:13.815816 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.815757    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 13 18:50:13.815816 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.815762    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 13 18:50:13.815816 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.815771    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 13 18:50:13.815816 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.815777    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 13 18:50:13.815948 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.815928    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 13 18:50:13.815948 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.815935    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 13 18:50:13.815948 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.815942    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 13 18:50:13.815948 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.815948    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 13 18:50:13.816053 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.815957    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 13 18:50:13.816053 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.815965    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 13 18:50:13.816583 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.816566    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 13 18:50:13.816628 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.816584    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 13 18:50:13.819570 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.819545    2565 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-128-215.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 13 18:50:13.819689 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:50:13.819669    2565 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-128-215.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 13 18:50:13.819754 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:50:13.819682    2565 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 13 18:50:13.820151 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.820136    2565 server.go:1268] "Started kubelet"
Jul 13 18:50:13.820334 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.820242    2565 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 13 18:50:13.820518 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.820300    2565 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 13 18:50:13.824205 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.824118    2565 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 13 18:50:13.824745 ip-10-0-128-215 systemd[1]: Started Kubernetes Kubelet.
Jul 13 18:50:13.826356 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.826299    2565 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 13 18:50:13.826659 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:50:13.826615    2565 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 13 18:50:13.826867 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.826850    2565 server.go:310] "Adding debug handlers to kubelet server"
Jul 13 18:50:13.831922 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:50:13.831027    2565 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-215.ec2.internal.18c1ee009d317abe  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-215.ec2.internal,UID:ip-10-0-128-215.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-128-215.ec2.internal,},FirstTimestamp:2026-07-13 18:50:13.820103358 +0000 UTC m=+0.345236768,LastTimestamp:2026-07-13 18:50:13.820103358 +0000 UTC m=+0.345236768,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-215.ec2.internal,}"
Jul 13 18:50:13.833248 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.833230    2565 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 13 18:50:13.833319 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.833251    2565 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 13 18:50:13.833420 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:50:13.833349    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-215.ec2.internal\" not found"
Jul 13 18:50:13.834076 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.833897    2565 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 13 18:50:13.834076 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.833915    2565 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 13 18:50:13.834076 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.833922    2565 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 13 18:50:13.834260 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.834085    2565 reconstruct.go:97] "Volume reconstruction finished"
Jul 13 18:50:13.834260 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.834096    2565 reconciler.go:29] "Reconciler: start to sync state"
Jul 13 18:50:13.834485 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.834467    2565 factory.go:153] Registering CRI-O factory
Jul 13 18:50:13.834485 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.834487    2565 factory.go:223] Registration of the crio container factory successfully
Jul 13 18:50:13.834628 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.834545    2565 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 13 18:50:13.834628 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.834555    2565 factory.go:55] Registering systemd factory
Jul 13 18:50:13.834628 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.834563    2565 factory.go:223] Registration of the systemd container factory successfully
Jul 13 18:50:13.834628 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.834583    2565 factory.go:103] Registering Raw factory
Jul 13 18:50:13.834628 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.834598    2565 manager.go:1196] Started watching for new ooms in manager
Jul 13 18:50:13.835058 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.835045    2565 manager.go:349] Starting recovery of all containers
Jul 13 18:50:13.841069 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.840848    2565 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-9mzhs"
Jul 13 18:50:13.843972 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:50:13.843946    2565 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 13 18:50:13.844108 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:50:13.844034    2565 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-128-215.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 13 18:50:13.845771 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.845748    2565 manager.go:354] Recovery completed
Jul 13 18:50:13.848790 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.848769    2565 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-9mzhs"
Jul 13 18:50:13.851492 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.851477    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 13 18:50:13.853659 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.853640    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-215.ec2.internal" event="NodeHasSufficientMemory"
Jul 13 18:50:13.853777 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.853674    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-215.ec2.internal" event="NodeHasNoDiskPressure"
Jul 13 18:50:13.853777 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.853685    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-215.ec2.internal" event="NodeHasSufficientPID"
Jul 13 18:50:13.854258 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.854240    2565 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 13 18:50:13.854258 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.854257    2565 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 13 18:50:13.854375 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.854268    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 13 18:50:13.854375 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.854275    2565 state_mem.go:36] "Initialized new in-memory state store"
Jul 13 18:50:13.855042 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.855027    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-215.ec2.internal" event="NodeHasSufficientMemory"
Jul 13 18:50:13.855117 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.855058    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-215.ec2.internal" event="NodeHasNoDiskPressure"
Jul 13 18:50:13.855117 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.855082    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-215.ec2.internal" event="NodeHasSufficientPID"
Jul 13 18:50:13.855478 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.855466    2565 policy_none.go:49] "None policy: Start"
Jul 13 18:50:13.855527 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.855483    2565 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 13 18:50:13.855527 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.855493    2565 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 13 18:50:13.856086 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:50:13.855998    2565 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-215.ec2.internal.18c1ee009f318468  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-215.ec2.internal,UID:ip-10-0-128-215.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-128-215.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-128-215.ec2.internal,},FirstTimestamp:2026-07-13 18:50:13.853660264 +0000 UTC m=+0.378793677,LastTimestamp:2026-07-13 18:50:13.853660264 +0000 UTC m=+0.378793677,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-215.ec2.internal,}"
Jul 13 18:50:13.856556 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.856545    2565 policy_none.go:47] "Start"
Jul 13 18:50:13.891486 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.891466    2565 manager.go:337] "Starting Device Plugin manager"
Jul 13 18:50:13.891613 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:50:13.891503    2565 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 13 18:50:13.891613 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.891515    2565 server.go:87] "Starting device plugin registration server"
Jul 13 18:50:13.891860 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.891847    2565 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 13 18:50:13.891910 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.891863    2565 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 13 18:50:13.891976 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.891956    2565 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 13 18:50:13.892053 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.892041    2565 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 13 18:50:13.892100 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.892054    2565 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 13 18:50:13.892636 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:50:13.892614    2565 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 13 18:50:13.892742 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:50:13.892672    2565 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-128-215.ec2.internal\" not found"
Jul 13 18:50:13.992997 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.992930    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 13 18:50:13.994654 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.994636    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-215.ec2.internal" event="NodeHasSufficientMemory"
Jul 13 18:50:13.994783 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.994672    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-215.ec2.internal" event="NodeHasNoDiskPressure"
Jul 13 18:50:13.994783 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.994686    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-215.ec2.internal" event="NodeHasSufficientPID"
Jul 13 18:50:13.994783 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:13.994733    2565 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-128-215.ec2.internal"
Jul 13 18:50:14.002534 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:14.002470    2565 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-128-215.ec2.internal"
Jul 13 18:50:14.002534 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:50:14.002502    2565 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-128-215.ec2.internal\": node \"ip-10-0-128-215.ec2.internal\" not found"
Jul 13 18:50:14.040019 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:14.039973    2565 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 13 18:50:14.041466 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:14.041436    2565 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 13 18:50:14.041588 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:14.041485    2565 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 13 18:50:14.041588 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:14.041516    2565 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 13 18:50:14.041588 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:50:14.041556    2565 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 13 18:50:14.050447 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:14.050420    2565 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 13 18:50:14.056174 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:50:14.056151    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-215.ec2.internal\" not found"
Jul 13 18:50:14.141696 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:14.141656    2565 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-215.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-128-215.ec2.internal"]
Jul 13 18:50:14.141891 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:14.141753    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 13 18:50:14.143460 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:14.143443    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-215.ec2.internal" event="NodeHasSufficientMemory"
Jul 13 18:50:14.143532 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:14.143475    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-215.ec2.internal" event="NodeHasNoDiskPressure"
Jul 13 18:50:14.143532 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:14.143486    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-215.ec2.internal" event="NodeHasSufficientPID"
Jul 13 18:50:14.144745 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:14.144732    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 13 18:50:14.144814 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:14.144798    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-215.ec2.internal"
Jul 13 18:50:14.144858 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:14.144832    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 13 18:50:14.145516 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:14.145500    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-215.ec2.internal" event="NodeHasSufficientMemory"
Jul 13 18:50:14.145580 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:14.145536    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-215.ec2.internal" event="NodeHasNoDiskPressure"
Jul 13 18:50:14.145580 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:14.145547    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-215.ec2.internal" event="NodeHasSufficientPID"
Jul 13 18:50:14.145646 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:14.145500    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-215.ec2.internal" event="NodeHasSufficientMemory"
Jul 13 18:50:14.145646 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:14.145613    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-215.ec2.internal" event="NodeHasNoDiskPressure"
Jul 13 18:50:14.145646 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:14.145626    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-215.ec2.internal" event="NodeHasSufficientPID"
Jul 13 18:50:14.146762 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:14.146743    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-215.ec2.internal"
Jul 13 18:50:14.146859 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:14.146770    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 13 18:50:14.147465 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:14.147449    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-215.ec2.internal" event="NodeHasSufficientMemory"
Jul 13 18:50:14.147546 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:14.147484    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-215.ec2.internal" event="NodeHasNoDiskPressure"
Jul 13 18:50:14.147546 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:14.147501    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-215.ec2.internal" event="NodeHasSufficientPID"
Jul 13 18:50:14.156235 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:50:14.156214    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-215.ec2.internal\" not found"
Jul 13 18:50:14.170557 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:50:14.170526    2565 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-215.ec2.internal\" not found" node="ip-10-0-128-215.ec2.internal"
Jul 13 18:50:14.174968 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:50:14.174949    2565 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-215.ec2.internal\" not found" node="ip-10-0-128-215.ec2.internal"
Jul 13 18:50:14.237384 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:14.237349    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/33d859ca02a99923eded717ac20e75f3-config\") pod \"kube-apiserver-proxy-ip-10-0-128-215.ec2.internal\" (UID: \"33d859ca02a99923eded717ac20e75f3\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-215.ec2.internal"
Jul 13 18:50:14.237384 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:14.237385    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/fce2836fb78b250794cad725b074cef4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-215.ec2.internal\" (UID: \"fce2836fb78b250794cad725b074cef4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-215.ec2.internal"
Jul 13 18:50:14.237542 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:14.237408    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fce2836fb78b250794cad725b074cef4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-215.ec2.internal\" (UID: \"fce2836fb78b250794cad725b074cef4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-215.ec2.internal"
Jul 13 18:50:14.256875 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:50:14.256808    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-215.ec2.internal\" not found"
Jul 13 18:50:14.338696 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:14.338659    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/33d859ca02a99923eded717ac20e75f3-config\") pod \"kube-apiserver-proxy-ip-10-0-128-215.ec2.internal\" (UID: \"33d859ca02a99923eded717ac20e75f3\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-215.ec2.internal"
Jul 13 18:50:14.338696 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:14.338696    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/fce2836fb78b250794cad725b074cef4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-215.ec2.internal\" (UID: \"fce2836fb78b250794cad725b074cef4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-215.ec2.internal"
Jul 13 18:50:14.338840 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:14.338737    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fce2836fb78b250794cad725b074cef4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-215.ec2.internal\" (UID: \"fce2836fb78b250794cad725b074cef4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-215.ec2.internal"
Jul 13 18:50:14.338840 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:14.338783    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fce2836fb78b250794cad725b074cef4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-215.ec2.internal\" (UID: \"fce2836fb78b250794cad725b074cef4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-215.ec2.internal"
Jul 13 18:50:14.338840 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:14.338786    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/33d859ca02a99923eded717ac20e75f3-config\") pod \"kube-apiserver-proxy-ip-10-0-128-215.ec2.internal\" (UID: \"33d859ca02a99923eded717ac20e75f3\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-215.ec2.internal"
Jul 13 18:50:14.338840 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:14.338786    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/fce2836fb78b250794cad725b074cef4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-215.ec2.internal\" (UID: \"fce2836fb78b250794cad725b074cef4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-215.ec2.internal"
Jul 13 18:50:14.357159 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:50:14.357108    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-215.ec2.internal\" not found"
Jul 13 18:50:14.458071 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:50:14.458027    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-215.ec2.internal\" not found"
Jul 13 18:50:14.473371 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:14.473343    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-215.ec2.internal"
Jul 13 18:50:14.477848 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:14.477626    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-215.ec2.internal"
Jul 13 18:50:14.559072 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:50:14.558965    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-215.ec2.internal\" not found"
Jul 13 18:50:14.644988 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:14.644960    2565 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 13 18:50:14.651682 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:14.651659    2565 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 13 18:50:14.659620 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:50:14.659585    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-215.ec2.internal\" not found"
Jul 13 18:50:14.760632 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:50:14.760595    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-215.ec2.internal\" not found"
Jul 13 18:50:14.771939 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:14.771906    2565 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 13 18:50:14.772110 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:14.772085    2565 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 13 18:50:14.772160 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:14.772123    2565 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 13 18:50:14.772160 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:14.772123    2565 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 13 18:50:14.853359 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:14.853247    2565 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-12 18:45:14 +0000 UTC" deadline="2028-04-29 11:20:07.664321544 +0000 UTC"
Jul 13 18:50:14.853359 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:14.853295    2565 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15736h29m52.811030516s"
Jul 13 18:50:14.861577 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:50:14.861534    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-215.ec2.internal\" not found"
Jul 13 18:50:14.961933 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:50:14.961895    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-215.ec2.internal\" not found"
Jul 13 18:50:15.038633 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:15.038593    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33d859ca02a99923eded717ac20e75f3.slice/crio-16cb9a8d62adf72c7663200b2199049fc76a3c6b4c391da5728488d4652b83e6 WatchSource:0}: Error finding container 16cb9a8d62adf72c7663200b2199049fc76a3c6b4c391da5728488d4652b83e6: Status 404 returned error can't find the container with id 16cb9a8d62adf72c7663200b2199049fc76a3c6b4c391da5728488d4652b83e6
Jul 13 18:50:15.043693 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:15.043674    2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 13 18:50:15.045035 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:15.044991    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-215.ec2.internal" event={"ID":"33d859ca02a99923eded717ac20e75f3","Type":"ContainerStarted","Data":"16cb9a8d62adf72c7663200b2199049fc76a3c6b4c391da5728488d4652b83e6"}
Jul 13 18:50:15.056152 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:15.056130    2565 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 13 18:50:15.138808 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:15.138687    2565 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-215.ec2.internal"
Jul 13 18:50:15.163840 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:15.163811    2565 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 13 18:50:15.165217 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:15.165196    2565 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-215.ec2.internal"
Jul 13 18:50:15.176074 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:15.176046    2565 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 13 18:50:15.811198 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:15.811148    2565 apiserver.go:52] "Watching apiserver"
Jul 13 18:50:15.823911 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:15.823878    2565 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 13 18:50:15.824081 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:15.823988    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfce2836fb78b250794cad725b074cef4.slice/crio-69756716edb28839a1ab069baa44914aacb1d9a6b227a7b481fb74cd9d0e31db WatchSource:0}: Error finding container 69756716edb28839a1ab069baa44914aacb1d9a6b227a7b481fb74cd9d0e31db: Status 404 returned error can't find the container with id 69756716edb28839a1ab069baa44914aacb1d9a6b227a7b481fb74cd9d0e31db
Jul 13 18:50:15.825742 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:15.825695    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/network-metrics-daemon-4sl67","openshift-network-operator/iptables-alerter-pxrrr","openshift-ovn-kubernetes/ovnkube-node-2bscc","kube-system/kube-apiserver-proxy-ip-10-0-128-215.ec2.internal","openshift-cluster-node-tuning-operator/tuned-gwvg9","openshift-dns/node-resolver-zcq5p","openshift-multus/multus-additional-cni-plugins-kgrbw","openshift-multus/multus-qtpwz","openshift-network-diagnostics/network-check-target-5tgm8","kube-system/global-pull-secret-syncer-qlzr5","kube-system/konnectivity-agent-n4lpd","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-psj7d","openshift-image-registry/node-ca-k2vg4","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-215.ec2.internal"]
Jul 13 18:50:15.827599 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:15.827573    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-n4lpd"
Jul 13 18:50:15.829935 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:15.829488    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-pxrrr"
Jul 13 18:50:15.830545 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:15.830513    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-2bscc"
Jul 13 18:50:15.831211 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:15.831189    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-rwd9x\""
Jul 13 18:50:15.831772 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:15.831714    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 13 18:50:15.832070 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:15.832052    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4sl67"
Jul 13 18:50:15.832160 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:50:15.832130    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4sl67" podUID="d4fd4b59-2ecd-497b-a485-1e94101a9033"
Jul 13 18:50:15.832632 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:15.832610    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 13 18:50:15.833435 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:15.833411    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-k2vg4"
Jul 13 18:50:15.834628 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:15.834606    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 13 18:50:15.834768 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:15.834650    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 13 18:50:15.835266 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:15.834875    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 13 18:50:15.835266 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:15.834887    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 13 18:50:15.835266 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:15.835037    2565 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 13 18:50:15.835266 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:15.835147    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-p699b\""
Jul 13 18:50:15.837730 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:15.835987    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 13 18:50:15.837730 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:15.836157    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 13 18:50:15.837730 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:15.836423    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 13 18:50:15.837730 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:15.836593    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 13 18:50:15.837730 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:15.837520    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-p96qv\""
Jul 13 18:50:15.838177 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:15.837844    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 13 18:50:15.838177 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:15.838086    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 13 18:50:15.838671 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:15.838414    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 13 18:50:15.838671 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:15.838569    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-7v5xt\""
Jul 13 18:50:15.838851 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:15.838680    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 13 18:50:15.839202 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:15.839181    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-zcq5p"
Jul 13 18:50:15.840912 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:15.840886    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-kgrbw"
Jul 13 18:50:15.842764 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:15.842744    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-qtpwz"
Jul 13 18:50:15.846017 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:15.845987    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qlzr5"
Jul 13 18:50:15.846150 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:50:15.846132    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qlzr5" podUID="6bbc0748-e539-4679-a100-1ba45794129b"
Jul 13 18:50:15.846643 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:15.846617    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 13 18:50:15.846716 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:15.846699    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 13 18:50:15.847121 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:15.846987    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2f0655bd-4a09-4314-8c1b-e30191eaec0e-run-openvswitch\") pod \"ovnkube-node-2bscc\" (UID: \"2f0655bd-4a09-4314-8c1b-e30191eaec0e\") " pod="openshift-ovn-kubernetes/ovnkube-node-2bscc"
Jul 13 18:50:15.847121 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:15.847035    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2f0655bd-4a09-4314-8c1b-e30191eaec0e-node-log\") pod \"ovnkube-node-2bscc\" (UID: \"2f0655bd-4a09-4314-8c1b-e30191eaec0e\") " pod="openshift-ovn-kubernetes/ovnkube-node-2bscc"
Jul 13 18:50:15.847299 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:15.847113    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b7240a5a-c704-49a7-b152-bdeb36cee9d4-host\") pod \"node-ca-k2vg4\" (UID: \"b7240a5a-c704-49a7-b152-bdeb36cee9d4\") " pod="openshift-image-registry/node-ca-k2vg4"
Jul 13 18:50:15.847299 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:15.847165    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b7240a5a-c704-49a7-b152-bdeb36cee9d4-serviceca\") pod \"node-ca-k2vg4\" (UID: \"b7240a5a-c704-49a7-b152-bdeb36cee9d4\") " pod="openshift-image-registry/node-ca-k2vg4"
Jul 13 18:50:15.847299 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:15.847187    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlt72\" (UniqueName: \"kubernetes.io/projected/b7240a5a-c704-49a7-b152-bdeb36cee9d4-kube-api-access-dlt72\") pod \"node-ca-k2vg4\" (UID: \"b7240a5a-c704-49a7-b152-bdeb36cee9d4\") " pod="openshift-image-registry/node-ca-k2vg4"
Jul 13 18:50:15.847299 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:15.847209    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6ae58b1e-3aaf-45dc-a741-2fb895468ca7-hosts-file\") pod \"node-resolver-zcq5p\" (UID: \"6ae58b1e-3aaf-45dc-a741-2fb895468ca7\") " pod="openshift-dns/node-resolver-zcq5p"
Jul 13 18:50:15.847299 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:15.847235    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2f0655bd-4a09-4314-8c1b-e30191eaec0e-host-slash\") pod \"ovnkube-node-2bscc\" (UID: \"2f0655bd-4a09-4314-8c1b-e30191eaec0e\") " pod="openshift-ovn-kubernetes/ovnkube-node-2bscc"
Jul 13 18:50:15.847299 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:15.847271    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2f0655bd-4a09-4314-8c1b-e30191eaec0e-host-run-netns\") pod \"ovnkube-node-2bscc\" (UID: \"2f0655bd-4a09-4314-8c1b-e30191eaec0e\") " pod="openshift-ovn-kubernetes/ovnkube-node-2bscc"
Jul 13 18:50:15.847591 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:15.847305    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2f0655bd-4a09-4314-8c1b-e30191eaec0e-host-cni-bin\") pod \"ovnkube-node-2bscc\" (UID: \"2f0655bd-4a09-4314-8c1b-e30191eaec0e\") " pod="openshift-ovn-kubernetes/ovnkube-node-2bscc"
Jul 13 18:50:15.847591 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:15.847367    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2f0655bd-4a09-4314-8c1b-e30191eaec0e-ovn-node-metrics-cert\") pod \"ovnkube-node-2bscc\" (UID: \"2f0655bd-4a09-4314-8c1b-e30191eaec0e\") " pod="openshift-ovn-kubernetes/ovnkube-node-2bscc"
Jul 13 18:50:15.847591 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:15.847417    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2f0655bd-4a09-4314-8c1b-e30191eaec0e-ovnkube-script-lib\") pod \"ovnkube-node-2bscc\" (UID: \"2f0655bd-4a09-4314-8c1b-e30191eaec0e\") " pod="openshift-ovn-kubernetes/ovnkube-node-2bscc"
Jul 13 18:50:15.847591 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:15.847455    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-9gbfj\""
Jul 13 18:50:15.847591 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:15.847479    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6s6s\" (UniqueName: \"kubernetes.io/projected/d4fd4b59-2ecd-497b-a485-1e94101a9033-kube-api-access-m6s6s\") pod \"network-metrics-daemon-4sl67\" (UID: \"d4fd4b59-2ecd-497b-a485-1e94101a9033\") " pod="openshift-multus/network-metrics-daemon-4sl67"
Jul 13 18:50:15.847591 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:15.847517    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dz2kb\" (UniqueName: \"kubernetes.io/projected/10b9cbe5-c0a3-4d27-af48-2cebde3572bc-kube-api-access-dz2kb\") pod \"iptables-alerter-pxrrr\" (UID: \"10b9cbe5-c0a3-4d27-af48-2cebde3572bc\") " pod="openshift-network-operator/iptables-alerter-pxrrr"
Jul 13 18:50:15.847591 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:15.847543    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdknc\" (UniqueName: \"kubernetes.io/projected/2f0655bd-4a09-4314-8c1b-e30191eaec0e-kube-api-access-xdknc\") pod \"ovnkube-node-2bscc\" (UID: \"2f0655bd-4a09-4314-8c1b-e30191eaec0e\") " pod="openshift-ovn-kubernetes/ovnkube-node-2bscc"
Jul 13 18:50:15.847938 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:15.847616    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9dccc391-865f-45a2-9004-85e3830f39be-konnectivity-ca\") pod \"konnectivity-agent-n4lpd\" (UID: \"9dccc391-865f-45a2-9004-85e3830f39be\") " pod="kube-system/konnectivity-agent-n4lpd"
Jul 13 18:50:15.847938 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:15.847649    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5tgm8"
Jul 13 18:50:15.847938 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:15.847653    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2f0655bd-4a09-4314-8c1b-e30191eaec0e-host-cni-netd\") pod \"ovnkube-node-2bscc\" (UID: \"2f0655bd-4a09-4314-8c1b-e30191eaec0e\") " pod="openshift-ovn-kubernetes/ovnkube-node-2bscc"
Jul 13 18:50:15.847938 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:50:15.847797    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5tgm8" podUID="76869c1f-1537-4e3e-b7e2-654a08cd2376"
Jul 13 18:50:15.847938 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:15.847877    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2f0655bd-4a09-4314-8c1b-e30191eaec0e-env-overrides\") pod \"ovnkube-node-2bscc\" (UID: \"2f0655bd-4a09-4314-8c1b-e30191eaec0e\") " pod="openshift-ovn-kubernetes/ovnkube-node-2bscc"
Jul 13 18:50:15.847938 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:15.847931    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9dccc391-865f-45a2-9004-85e3830f39be-agent-certs\") pod \"konnectivity-agent-n4lpd\" (UID: \"9dccc391-865f-45a2-9004-85e3830f39be\") " pod="kube-system/konnectivity-agent-n4lpd"
Jul 13 18:50:15.848249 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:15.847973    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/10b9cbe5-c0a3-4d27-af48-2cebde3572bc-host-slash\") pod \"iptables-alerter-pxrrr\" (UID: \"10b9cbe5-c0a3-4d27-af48-2cebde3572bc\") " pod="openshift-network-operator/iptables-alerter-pxrrr"
Jul 13 18:50:15.848249 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:15.848024    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2f0655bd-4a09-4314-8c1b-e30191eaec0e-systemd-units\") pod \"ovnkube-node-2bscc\" (UID: \"2f0655bd-4a09-4314-8c1b-e30191eaec0e\") " pod="openshift-ovn-kubernetes/ovnkube-node-2bscc"
Jul 13 18:50:15.848249 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:15.848062    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2f0655bd-4a09-4314-8c1b-e30191eaec0e-run-ovn\") pod \"ovnkube-node-2bscc\" (UID: \"2f0655bd-4a09-4314-8c1b-e30191eaec0e\") " pod="openshift-ovn-kubernetes/ovnkube-node-2bscc"
Jul 13 18:50:15.848249 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:15.848097    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2f0655bd-4a09-4314-8c1b-e30191eaec0e-host-run-ovn-kubernetes\") pod \"ovnkube-node-2bscc\" (UID: \"2f0655bd-4a09-4314-8c1b-e30191eaec0e\") " pod="openshift-ovn-kubernetes/ovnkube-node-2bscc"
Jul 13 18:50:15.848249 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:15.848139    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2f0655bd-4a09-4314-8c1b-e30191eaec0e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2bscc\" (UID: \"2f0655bd-4a09-4314-8c1b-e30191eaec0e\") " pod="openshift-ovn-kubernetes/ovnkube-node-2bscc"
Jul 13 18:50:15.848249 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:15.848186    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/10b9cbe5-c0a3-4d27-af48-2cebde3572bc-iptables-alerter-script\") pod \"iptables-alerter-pxrrr\" (UID: \"10b9cbe5-c0a3-4d27-af48-2cebde3572bc\") " pod="openshift-network-operator/iptables-alerter-pxrrr"
Jul 13 18:50:15.848249 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:15.848225    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2f0655bd-4a09-4314-8c1b-e30191eaec0e-var-lib-openvswitch\") pod \"ovnkube-node-2bscc\" (UID: \"2f0655bd-4a09-4314-8c1b-e30191eaec0e\") " pod="openshift-ovn-kubernetes/ovnkube-node-2bscc"
Jul 13 18:50:15.848592 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:15.848263    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2f0655bd-4a09-4314-8c1b-e30191eaec0e-etc-openvswitch\") pod \"ovnkube-node-2bscc\" (UID: \"2f0655bd-4a09-4314-8c1b-e30191eaec0e\") " pod="openshift-ovn-kubernetes/ovnkube-node-2bscc"
Jul 13 18:50:15.848592 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:15.848296    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2f0655bd-4a09-4314-8c1b-e30191eaec0e-log-socket\") pod \"ovnkube-node-2bscc\" (UID: \"2f0655bd-4a09-4314-8c1b-e30191eaec0e\") " pod="openshift-ovn-kubernetes/ovnkube-node-2bscc"
Jul 13 18:50:15.848592 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:15.848334    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d4fd4b59-2ecd-497b-a485-1e94101a9033-metrics-certs\") pod \"network-metrics-daemon-4sl67\" (UID: \"d4fd4b59-2ecd-497b-a485-1e94101a9033\") " pod="openshift-multus/network-metrics-daemon-4sl67"
Jul 13 18:50:15.848592 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:15.848372    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2f0655bd-4a09-4314-8c1b-e30191eaec0e-host-kubelet\") pod \"ovnkube-node-2bscc\" (UID: \"2f0655bd-4a09-4314-8c1b-e30191eaec0e\") " pod="openshift-ovn-kubernetes/ovnkube-node-2bscc"
Jul 13 18:50:15.848592 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:15.848411    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2f0655bd-4a09-4314-8c1b-e30191eaec0e-run-systemd\") pod \"ovnkube-node-2bscc\" (UID: \"2f0655bd-4a09-4314-8c1b-e30191eaec0e\") " pod="openshift-ovn-kubernetes/ovnkube-node-2bscc"
Jul 13 18:50:15.848592 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:15.848455    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2f0655bd-4a09-4314-8c1b-e30191eaec0e-ovnkube-config\") pod \"ovnkube-node-2bscc\" (UID: \"2f0655bd-4a09-4314-8c1b-e30191eaec0e\") " pod="openshift-ovn-kubernetes/ovnkube-node-2bscc"
Jul 13 18:50:15.848592 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:15.848491    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ae58b1e-3aaf-45dc-a741-2fb895468ca7-tmp-dir\") pod \"node-resolver-zcq5p\" (UID: \"6ae58b1e-3aaf-45dc-a741-2fb895468ca7\") " pod="openshift-dns/node-resolver-zcq5p"
Jul 13 18:50:15.848592 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:15.848532    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmd5c\" (UniqueName: \"kubernetes.io/projected/6ae58b1e-3aaf-45dc-a741-2fb895468ca7-kube-api-access-rmd5c\") pod \"node-resolver-zcq5p\" (UID: \"6ae58b1e-3aaf-45dc-a741-2fb895468ca7\") " pod="openshift-dns/node-resolver-zcq5p"
Jul 13 18:50:15.848592 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:15.848554    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 13 18:50:15.848592 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:15.848592    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 13 18:50:15.849191 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:15.849167    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 13 18:50:15.849353 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:15.849336    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-47xtd\""
Jul 13 18:50:15.849419 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:15.849371    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 13 18:50:15.850118 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:15.850095    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-psj7d"
Jul 13 18:50:15.851056 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:15.850978    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-qgxdh\""
Jul 13 18:50:15.851634 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:15.851487    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 13 18:50:15.852753 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:15.852733    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 13 18:50:15.853686 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:15.853667    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-gwvg9"
Jul 13 18:50:15.863737 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:15.863701    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 13 18:50:15.863903 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:15.863853    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 13 18:50:15.864012 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:15.863943    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 13 18:50:15.864156 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:15.864137    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-8ght8\""
Jul 13 18:50:15.864886 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:15.864859    2565 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 13 18:50:15.865335 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:15.865315    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-9prxv\""
Jul 13 18:50:15.865539 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:15.865318    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 13 18:50:15.865632 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:15.865402    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 13 18:50:15.903557 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:15.903522    2565 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-b5lg4"
Jul 13 18:50:15.913574 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:15.913538    2565 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-b5lg4"
Jul 13 18:50:15.935199 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:15.935122    2565 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 13 18:50:15.949932 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:15.949893    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/10b9cbe5-c0a3-4d27-af48-2cebde3572bc-host-slash\") pod \"iptables-alerter-pxrrr\" (UID: \"10b9cbe5-c0a3-4d27-af48-2cebde3572bc\") " pod="openshift-network-operator/iptables-alerter-pxrrr"
Jul 13 18:50:15.949932 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:15.949944    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2f0655bd-4a09-4314-8c1b-e30191eaec0e-systemd-units\") pod \"ovnkube-node-2bscc\" (UID: \"2f0655bd-4a09-4314-8c1b-e30191eaec0e\") " pod="openshift-ovn-kubernetes/ovnkube-node-2bscc"
Jul 13 18:50:15.950149 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:15.949975    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2f0655bd-4a09-4314-8c1b-e30191eaec0e-run-ovn\") pod \"ovnkube-node-2bscc\" (UID: \"2f0655bd-4a09-4314-8c1b-e30191eaec0e\") " pod="openshift-ovn-kubernetes/ovnkube-node-2bscc"
Jul 13 18:50:15.950149 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:15.950000    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/10b9cbe5-c0a3-4d27-af48-2cebde3572bc-host-slash\") pod \"iptables-alerter-pxrrr\" (UID: \"10b9cbe5-c0a3-4d27-af48-2cebde3572bc\") " pod="openshift-network-operator/iptables-alerter-pxrrr"
Jul 13 18:50:15.950149 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:15.950011    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2f0655bd-4a09-4314-8c1b-e30191eaec0e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2bscc\" (UID: \"2f0655bd-4a09-4314-8c1b-e30191eaec0e\") " pod="openshift-ovn-kubernetes/ovnkube-node-2bscc"
Jul 13 18:50:15.950149 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:15.950044    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ef958600-65d1-473a-b2dd-397c1dbbc8da-system-cni-dir\") pod \"multus-qtpwz\" (UID: \"ef958600-65d1-473a-b2dd-397c1dbbc8da\") " pod="openshift-multus/multus-qtpwz"
Jul 13 18:50:15.950149 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:15.950064    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2f0655bd-4a09-4314-8c1b-e30191eaec0e-systemd-units\") pod \"ovnkube-node-2bscc\" (UID: \"2f0655bd-4a09-4314-8c1b-e30191eaec0e\") " pod="openshift-ovn-kubernetes/ovnkube-node-2bscc"
Jul 13 18:50:15.950149 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:15.950080    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2f0655bd-4a09-4314-8c1b-e30191eaec0e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2bscc\" (UID: \"2f0655bd-4a09-4314-8c1b-e30191eaec0e\") " pod="openshift-ovn-kubernetes/ovnkube-node-2bscc"
Jul 13 18:50:15.950149 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:15.950098    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2f0655bd-4a09-4314-8c1b-e30191eaec0e-run-ovn\") pod \"ovnkube-node-2bscc\" (UID: \"2f0655bd-4a09-4314-8c1b-e30191eaec0e\") " pod="openshift-ovn-kubernetes/ovnkube-node-2bscc"
Jul 13 18:50:15.950149 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:15.950131    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ef958600-65d1-473a-b2dd-397c1dbbc8da-host-run-k8s-cni-cncf-io\") pod \"multus-qtpwz\" (UID: \"ef958600-65d1-473a-b2dd-397c1dbbc8da\") " pod="openshift-multus/multus-qtpwz"
Jul 13 18:50:15.950467 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:15.950162    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ef958600-65d1-473a-b2dd-397c1dbbc8da-hostroot\") pod \"multus-qtpwz\" (UID: \"ef958600-65d1-473a-b2dd-397c1dbbc8da\") " pod="openshift-multus/multus-qtpwz"
Jul 13 18:50:15.950467 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:15.950209    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2f0655bd-4a09-4314-8c1b-e30191eaec0e-etc-openvswitch\") pod \"ovnkube-node-2bscc\" (UID: \"2f0655bd-4a09-4314-8c1b-e30191eaec0e\") " pod="openshift-ovn-kubernetes/ovnkube-node-2bscc"
Jul 13 18:50:15.950467 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:15.950240    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ef958600-65d1-473a-b2dd-397c1dbbc8da-cnibin\") pod \"multus-qtpwz\" (UID: \"ef958600-65d1-473a-b2dd-397c1dbbc8da\") " pod="openshift-multus/multus-qtpwz"
Jul 13 18:50:15.950467 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:15.950266    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dxwr\" (UniqueName: \"kubernetes.io/projected/ef958600-65d1-473a-b2dd-397c1dbbc8da-kube-api-access-9dxwr\") pod \"multus-qtpwz\" (UID: \"ef958600-65d1-473a-b2dd-397c1dbbc8da\") " pod="openshift-multus/multus-qtpwz"
Jul 13 18:50:15.950467 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:15.950272    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2f0655bd-4a09-4314-8c1b-e30191eaec0e-etc-openvswitch\") pod \"ovnkube-node-2bscc\" (UID: \"2f0655bd-4a09-4314-8c1b-e30191eaec0e\") " pod="openshift-ovn-kubernetes/ovnkube-node-2bscc"
Jul 13 18:50:15.950467 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:15.950296    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6bbc0748-e539-4679-a100-1ba45794129b-original-pull-secret\") pod \"global-pull-secret-syncer-qlzr5\" (UID: \"6bbc0748-e539-4679-a100-1ba45794129b\") " pod="kube-system/global-pull-secret-syncer-qlzr5"
Jul 13 18:50:15.950467 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:15.950317    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3a32830b-434d-4118-a91c-9dfdd29dcfa6-sys\") pod \"tuned-gwvg9\" (UID: \"3a32830b-434d-4118-a91c-9dfdd29dcfa6\") " pod="openshift-cluster-node-tuning-operator/tuned-gwvg9"
Jul 13 18:50:15.950467 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:15.950338    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2f0655bd-4a09-4314-8c1b-e30191eaec0e-host-kubelet\") pod \"ovnkube-node-2bscc\" (UID: \"2f0655bd-4a09-4314-8c1b-e30191eaec0e\") " pod="openshift-ovn-kubernetes/ovnkube-node-2bscc"
Jul 13 18:50:15.950467 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:15.950366    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2f0655bd-4a09-4314-8c1b-e30191eaec0e-run-systemd\") pod \"ovnkube-node-2bscc\" (UID: \"2f0655bd-4a09-4314-8c1b-e30191eaec0e\") " pod="openshift-ovn-kubernetes/ovnkube-node-2bscc"
Jul 13 18:50:15.950467 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:15.950381    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2f0655bd-4a09-4314-8c1b-e30191eaec0e-ovnkube-config\") pod \"ovnkube-node-2bscc\" (UID: \"2f0655bd-4a09-4314-8c1b-e30191eaec0e\") " pod="openshift-ovn-kubernetes/ovnkube-node-2bscc"
Jul 13 18:50:15.950467 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:15.950396    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ae58b1e-3aaf-45dc-a741-2fb895468ca7-tmp-dir\") pod \"node-resolver-zcq5p\" (UID: \"6ae58b1e-3aaf-45dc-a741-2fb895468ca7\") " pod="openshift-dns/node-resolver-zcq5p"
Jul 13 18:50:15.950467 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:15.950413    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmd5c\" (UniqueName: \"kubernetes.io/projected/6ae58b1e-3aaf-45dc-a741-2fb895468ca7-kube-api-access-rmd5c\") pod \"node-resolver-zcq5p\" (UID: \"6ae58b1e-3aaf-45dc-a741-2fb895468ca7\") " pod="openshift-dns/node-resolver-zcq5p"
Jul 13 18:50:15.950467 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:15.950430    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6ca2c2d9-346a-48c4-af31-ab3b2e66539f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-kgrbw\" (UID: \"6ca2c2d9-346a-48c4-af31-ab3b2e66539f\") " pod="openshift-multus/multus-additional-cni-plugins-kgrbw"
Jul 13 18:50:15.950467 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:15.950432    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2f0655bd-4a09-4314-8c1b-e30191eaec0e-host-kubelet\") pod \"ovnkube-node-2bscc\" (UID: \"2f0655bd-4a09-4314-8c1b-e30191eaec0e\") " pod="openshift-ovn-kubernetes/ovnkube-node-2bscc"
Jul 13 18:50:15.950467 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:15.950448    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ef958600-65d1-473a-b2dd-397c1dbbc8da-host-run-multus-certs\") pod \"multus-qtpwz\" (UID: \"ef958600-65d1-473a-b2dd-397c1dbbc8da\") " pod="openshift-multus/multus-qtpwz"
Jul 13 18:50:15.950467 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:15.950465    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3a32830b-434d-4118-a91c-9dfdd29dcfa6-etc-modprobe-d\") pod \"tuned-gwvg9\" (UID: \"3a32830b-434d-4118-a91c-9dfdd29dcfa6\") " pod="openshift-cluster-node-tuning-operator/tuned-gwvg9"
Jul 13 18:50:15.951333 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:15.950486    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2f0655bd-4a09-4314-8c1b-e30191eaec0e-run-systemd\") pod \"ovnkube-node-2bscc\" (UID: \"2f0655bd-4a09-4314-8c1b-e30191eaec0e\") " pod="openshift-ovn-kubernetes/ovnkube-node-2bscc"
Jul 13 18:50:15.951333 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:15.950502    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2f0655bd-4a09-4314-8c1b-e30191eaec0e-node-log\") pod \"ovnkube-node-2bscc\" (UID: \"2f0655bd-4a09-4314-8c1b-e30191eaec0e\") " pod="openshift-ovn-kubernetes/ovnkube-node-2bscc"
Jul 13 18:50:15.951333 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:15.950543    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2f0655bd-4a09-4314-8c1b-e30191eaec0e-node-log\") pod \"ovnkube-node-2bscc\" (UID: \"2f0655bd-4a09-4314-8c1b-e30191eaec0e\") " pod="openshift-ovn-kubernetes/ovnkube-node-2bscc"
Jul 13 18:50:15.951333 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:15.950648    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b7240a5a-c704-49a7-b152-bdeb36cee9d4-serviceca\") pod \"node-ca-k2vg4\" (UID: \"b7240a5a-c704-49a7-b152-bdeb36cee9d4\") " pod="openshift-image-registry/node-ca-k2vg4"
Jul 13 18:50:15.951333 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:15.950692    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6ae58b1e-3aaf-45dc-a741-2fb895468ca7-hosts-file\") pod \"node-resolver-zcq5p\" (UID: \"6ae58b1e-3aaf-45dc-a741-2fb895468ca7\") " pod="openshift-dns/node-resolver-zcq5p"
Jul 13 18:50:15.951333 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:15.950765    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ae58b1e-3aaf-45dc-a741-2fb895468ca7-tmp-dir\") pod \"node-resolver-zcq5p\" (UID: \"6ae58b1e-3aaf-45dc-a741-2fb895468ca7\") " pod="openshift-dns/node-resolver-zcq5p"
Jul 13 18:50:15.951333 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:15.950774    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6ae58b1e-3aaf-45dc-a741-2fb895468ca7-hosts-file\") pod \"node-resolver-zcq5p\" (UID: \"6ae58b1e-3aaf-45dc-a741-2fb895468ca7\") " pod="openshift-dns/node-resolver-zcq5p"
Jul 13 18:50:15.951333 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:15.950814    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ef958600-65d1-473a-b2dd-397c1dbbc8da-multus-socket-dir-parent\") pod \"multus-qtpwz\" (UID: \"ef958600-65d1-473a-b2dd-397c1dbbc8da\") " pod="openshift-multus/multus-qtpwz"
Jul 13 18:50:15.951333 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:15.950872    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3a32830b-434d-4118-a91c-9dfdd29dcfa6-etc-kubernetes\") pod \"tuned-gwvg9\" (UID: \"3a32830b-434d-4118-a91c-9dfdd29dcfa6\") " pod="openshift-cluster-node-tuning-operator/tuned-gwvg9"
Jul 13 18:50:15.951333 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:15.950897    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3a32830b-434d-4118-a91c-9dfdd29dcfa6-run\") pod \"tuned-gwvg9\" (UID: \"3a32830b-434d-4118-a91c-9dfdd29dcfa6\") " pod="openshift-cluster-node-tuning-operator/tuned-gwvg9"
Jul 13 18:50:15.951333 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:15.950936    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xbqs\" (UniqueName: \"kubernetes.io/projected/3a32830b-434d-4118-a91c-9dfdd29dcfa6-kube-api-access-5xbqs\") pod \"tuned-gwvg9\" (UID: \"3a32830b-434d-4118-a91c-9dfdd29dcfa6\") " pod="openshift-cluster-node-tuning-operator/tuned-gwvg9"
Jul 13 18:50:15.951333 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:15.950961    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/6ca2c2d9-346a-48c4-af31-ab3b2e66539f-os-release\") pod \"multus-additional-cni-plugins-kgrbw\" (UID: \"6ca2c2d9-346a-48c4-af31-ab3b2e66539f\") " pod="openshift-multus/multus-additional-cni-plugins-kgrbw"
Jul 13 18:50:15.951333 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:15.950985    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3a32830b-434d-4118-a91c-9dfdd29dcfa6-lib-modules\") pod \"tuned-gwvg9\" (UID: \"3a32830b-434d-4118-a91c-9dfdd29dcfa6\") " pod="openshift-cluster-node-tuning-operator/tuned-gwvg9"
Jul 13 18:50:15.951333 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:15.951009    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ef958600-65d1-473a-b2dd-397c1dbbc8da-os-release\") pod \"multus-qtpwz\" (UID: \"ef958600-65d1-473a-b2dd-397c1dbbc8da\") " pod="openshift-multus/multus-qtpwz"
Jul 13 18:50:15.951333 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:15.951050    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ef958600-65d1-473a-b2dd-397c1dbbc8da-host-var-lib-kubelet\") pod \"multus-qtpwz\" (UID: \"ef958600-65d1-473a-b2dd-397c1dbbc8da\") " pod="openshift-multus/multus-qtpwz"
Jul 13 18:50:15.951333 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:15.951085    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/34d8b129-0c9e-4cda-8669-d7595f3d01af-device-dir\") pod \"aws-ebs-csi-driver-node-psj7d\" (UID: \"34d8b129-0c9e-4cda-8669-d7595f3d01af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-psj7d"
Jul 13 18:50:15.951333 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:15.951133    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2f0655bd-4a09-4314-8c1b-e30191eaec0e-ovnkube-config\") pod \"ovnkube-node-2bscc\" (UID: \"2f0655bd-4a09-4314-8c1b-e30191eaec0e\") " pod="openshift-ovn-kubernetes/ovnkube-node-2bscc"
Jul 13 18:50:15.952186 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:15.951140    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdknc\" (UniqueName: \"kubernetes.io/projected/2f0655bd-4a09-4314-8c1b-e30191eaec0e-kube-api-access-xdknc\") pod \"ovnkube-node-2bscc\" (UID: \"2f0655bd-4a09-4314-8c1b-e30191eaec0e\") " pod="openshift-ovn-kubernetes/ovnkube-node-2bscc"
Jul 13 18:50:15.952186 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:15.951175    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b7240a5a-c704-49a7-b152-bdeb36cee9d4-serviceca\") pod \"node-ca-k2vg4\" (UID: \"b7240a5a-c704-49a7-b152-bdeb36cee9d4\") " pod="openshift-image-registry/node-ca-k2vg4"
Jul 13 18:50:15.952186 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:15.951193    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9dccc391-865f-45a2-9004-85e3830f39be-konnectivity-ca\") pod \"konnectivity-agent-n4lpd\" (UID: \"9dccc391-865f-45a2-9004-85e3830f39be\") " pod="kube-system/konnectivity-agent-n4lpd"
Jul 13 18:50:15.952186 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:15.951221    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ef958600-65d1-473a-b2dd-397c1dbbc8da-host-run-netns\") pod \"multus-qtpwz\" (UID: \"ef958600-65d1-473a-b2dd-397c1dbbc8da\") " pod="openshift-multus/multus-qtpwz"
Jul 13 18:50:15.952186 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:15.951248    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ef958600-65d1-473a-b2dd-397c1dbbc8da-host-var-lib-cni-multus\") pod \"multus-qtpwz\" (UID: \"ef958600-65d1-473a-b2dd-397c1dbbc8da\") " pod="openshift-multus/multus-qtpwz"
Jul 13 18:50:15.952186 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:15.951276    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ef958600-65d1-473a-b2dd-397c1dbbc8da-multus-conf-dir\") pod \"multus-qtpwz\" (UID: \"ef958600-65d1-473a-b2dd-397c1dbbc8da\") " pod="openshift-multus/multus-qtpwz"
Jul 13 18:50:15.952186 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:15.951332    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ef958600-65d1-473a-b2dd-397c1dbbc8da-etc-kubernetes\") pod \"multus-qtpwz\" (UID: \"ef958600-65d1-473a-b2dd-397c1dbbc8da\") " pod="openshift-multus/multus-qtpwz"
Jul 13 18:50:15.952186 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:15.951356    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3a32830b-434d-4118-a91c-9dfdd29dcfa6-etc-tuned\") pod \"tuned-gwvg9\" (UID: \"3a32830b-434d-4118-a91c-9dfdd29dcfa6\") " pod="openshift-cluster-node-tuning-operator/tuned-gwvg9"
Jul 13 18:50:15.952186 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:15.951380    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvvhf\" (UniqueName: \"kubernetes.io/projected/76869c1f-1537-4e3e-b7e2-654a08cd2376-kube-api-access-qvvhf\") pod \"network-check-target-5tgm8\" (UID: \"76869c1f-1537-4e3e-b7e2-654a08cd2376\") " pod="openshift-network-diagnostics/network-check-target-5tgm8"
Jul 13 18:50:15.952186 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:15.951414    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2f0655bd-4a09-4314-8c1b-e30191eaec0e-host-cni-netd\") pod \"ovnkube-node-2bscc\" (UID: \"2f0655bd-4a09-4314-8c1b-e30191eaec0e\") " pod="openshift-ovn-kubernetes/ovnkube-node-2bscc"
Jul 13 18:50:15.952186 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:15.951434    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2f0655bd-4a09-4314-8c1b-e30191eaec0e-env-overrides\") pod \"ovnkube-node-2bscc\" (UID: \"2f0655bd-4a09-4314-8c1b-e30191eaec0e\") " pod="openshift-ovn-kubernetes/ovnkube-node-2bscc"
Jul 13 18:50:15.952186 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:15.951472    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3a32830b-434d-4118-a91c-9dfdd29dcfa6-etc-sysconfig\") pod \"tuned-gwvg9\" (UID: \"3a32830b-434d-4118-a91c-9dfdd29dcfa6\") " pod="openshift-cluster-node-tuning-operator/tuned-gwvg9"
Jul 13 18:50:15.952186 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:15.951566    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2f0655bd-4a09-4314-8c1b-e30191eaec0e-host-run-ovn-kubernetes\") pod \"ovnkube-node-2bscc\" (UID: \"2f0655bd-4a09-4314-8c1b-e30191eaec0e\") " pod="openshift-ovn-kubernetes/ovnkube-node-2bscc"
Jul 13 18:50:15.952186 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:15.951608    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6ca2c2d9-346a-48c4-af31-ab3b2e66539f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-kgrbw\" (UID: \"6ca2c2d9-346a-48c4-af31-ab3b2e66539f\") " pod="openshift-multus/multus-additional-cni-plugins-kgrbw"
Jul 13 18:50:15.952186 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:15.951667    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3a32830b-434d-4118-a91c-9dfdd29dcfa6-host\") pod \"tuned-gwvg9\" (UID: \"3a32830b-434d-4118-a91c-9dfdd29dcfa6\") " pod="openshift-cluster-node-tuning-operator/tuned-gwvg9"
Jul 13 18:50:15.952186 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:15.951700    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9dccc391-865f-45a2-9004-85e3830f39be-konnectivity-ca\") pod \"konnectivity-agent-n4lpd\" (UID: \"9dccc391-865f-45a2-9004-85e3830f39be\") " pod="kube-system/konnectivity-agent-n4lpd"
Jul 13 18:50:15.952186 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:15.951779    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/10b9cbe5-c0a3-4d27-af48-2cebde3572bc-iptables-alerter-script\") pod \"iptables-alerter-pxrrr\" (UID: \"10b9cbe5-c0a3-4d27-af48-2cebde3572bc\") " pod="openshift-network-operator/iptables-alerter-pxrrr"
Jul 13 18:50:15.953029 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:15.951822    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2f0655bd-4a09-4314-8c1b-e30191eaec0e-host-run-ovn-kubernetes\") pod \"ovnkube-node-2bscc\" (UID: \"2f0655bd-4a09-4314-8c1b-e30191eaec0e\") " pod="openshift-ovn-kubernetes/ovnkube-node-2bscc"
Jul 13 18:50:15.953029 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:15.951830    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2f0655bd-4a09-4314-8c1b-e30191eaec0e-var-lib-openvswitch\") pod \"ovnkube-node-2bscc\" (UID: \"2f0655bd-4a09-4314-8c1b-e30191eaec0e\") " pod="openshift-ovn-kubernetes/ovnkube-node-2bscc"
Jul 13 18:50:15.953029 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:15.951879    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2f0655bd-4a09-4314-8c1b-e30191eaec0e-log-socket\") pod \"ovnkube-node-2bscc\" (UID: \"2f0655bd-4a09-4314-8c1b-e30191eaec0e\") " pod="openshift-ovn-kubernetes/ovnkube-node-2bscc"
Jul 13 18:50:15.953029 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:15.951906    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d4fd4b59-2ecd-497b-a485-1e94101a9033-metrics-certs\") pod \"network-metrics-daemon-4sl67\" (UID: \"d4fd4b59-2ecd-497b-a485-1e94101a9033\") " pod="openshift-multus/network-metrics-daemon-4sl67"
Jul 13 18:50:15.953029 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:15.951918    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2f0655bd-4a09-4314-8c1b-e30191eaec0e-env-overrides\") pod \"ovnkube-node-2bscc\" (UID: \"2f0655bd-4a09-4314-8c1b-e30191eaec0e\") " pod="openshift-ovn-kubernetes/ovnkube-node-2bscc"
Jul 13 18:50:15.953029 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:15.951945    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kx558\" (UniqueName: \"kubernetes.io/projected/6ca2c2d9-346a-48c4-af31-ab3b2e66539f-kube-api-access-kx558\") pod \"multus-additional-cni-plugins-kgrbw\" (UID: \"6ca2c2d9-346a-48c4-af31-ab3b2e66539f\") " pod="openshift-multus/multus-additional-cni-plugins-kgrbw"
Jul 13 18:50:15.953029 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:15.951987    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2f0655bd-4a09-4314-8c1b-e30191eaec0e-var-lib-openvswitch\") pod \"ovnkube-node-2bscc\" (UID: \"2f0655bd-4a09-4314-8c1b-e30191eaec0e\") " pod="openshift-ovn-kubernetes/ovnkube-node-2bscc"
Jul 13 18:50:15.953029 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:15.952035    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ef958600-65d1-473a-b2dd-397c1dbbc8da-multus-cni-dir\") pod \"multus-qtpwz\" (UID: \"ef958600-65d1-473a-b2dd-397c1dbbc8da\") " pod="openshift-multus/multus-qtpwz"
Jul 13 18:50:15.953029 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:15.952069    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ef958600-65d1-473a-b2dd-397c1dbbc8da-multus-daemon-config\") pod \"multus-qtpwz\" (UID: \"ef958600-65d1-473a-b2dd-397c1dbbc8da\") " pod="openshift-multus/multus-qtpwz"
Jul 13 18:50:15.953029 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:15.952100    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pz4z9\" (UniqueName: \"kubernetes.io/projected/34d8b129-0c9e-4cda-8669-d7595f3d01af-kube-api-access-pz4z9\") pod \"aws-ebs-csi-driver-node-psj7d\" (UID: \"34d8b129-0c9e-4cda-8669-d7595f3d01af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-psj7d"
Jul 13 18:50:15.953029 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:15.952154    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2f0655bd-4a09-4314-8c1b-e30191eaec0e-log-socket\") pod \"ovnkube-node-2bscc\" (UID: \"2f0655bd-4a09-4314-8c1b-e30191eaec0e\") " pod="openshift-ovn-kubernetes/ovnkube-node-2bscc"
Jul 13 18:50:15.953029 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:15.952251    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2f0655bd-4a09-4314-8c1b-e30191eaec0e-host-cni-netd\") pod \"ovnkube-node-2bscc\" (UID: \"2f0655bd-4a09-4314-8c1b-e30191eaec0e\") " pod="openshift-ovn-kubernetes/ovnkube-node-2bscc"
Jul 13 18:50:15.953029 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:15.952298    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2f0655bd-4a09-4314-8c1b-e30191eaec0e-run-openvswitch\") pod \"ovnkube-node-2bscc\" (UID: \"2f0655bd-4a09-4314-8c1b-e30191eaec0e\") " pod="openshift-ovn-kubernetes/ovnkube-node-2bscc"
Jul 13 18:50:15.953029 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:15.952351    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b7240a5a-c704-49a7-b152-bdeb36cee9d4-host\") pod \"node-ca-k2vg4\" (UID: \"b7240a5a-c704-49a7-b152-bdeb36cee9d4\") " pod="openshift-image-registry/node-ca-k2vg4"
Jul 13 18:50:15.953029 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:50:15.952365    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 13 18:50:15.953029 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:15.952372    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/10b9cbe5-c0a3-4d27-af48-2cebde3572bc-iptables-alerter-script\") pod \"iptables-alerter-pxrrr\" (UID: \"10b9cbe5-c0a3-4d27-af48-2cebde3572bc\") " pod="openshift-network-operator/iptables-alerter-pxrrr"
Jul 13 18:50:15.953029 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:15.952378    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlt72\" (UniqueName: \"kubernetes.io/projected/b7240a5a-c704-49a7-b152-bdeb36cee9d4-kube-api-access-dlt72\") pod \"node-ca-k2vg4\" (UID: \"b7240a5a-c704-49a7-b152-bdeb36cee9d4\") " pod="openshift-image-registry/node-ca-k2vg4"
Jul 13 18:50:15.953835 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:50:15.952505    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d4fd4b59-2ecd-497b-a485-1e94101a9033-metrics-certs podName:d4fd4b59-2ecd-497b-a485-1e94101a9033 nodeName:}" failed. No retries permitted until 2026-07-13 18:50:16.452470049 +0000 UTC m=+2.977603457 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d4fd4b59-2ecd-497b-a485-1e94101a9033-metrics-certs") pod "network-metrics-daemon-4sl67" (UID: "d4fd4b59-2ecd-497b-a485-1e94101a9033") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 13 18:50:15.953835 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:15.952532    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2f0655bd-4a09-4314-8c1b-e30191eaec0e-run-openvswitch\") pod \"ovnkube-node-2bscc\" (UID: \"2f0655bd-4a09-4314-8c1b-e30191eaec0e\") " pod="openshift-ovn-kubernetes/ovnkube-node-2bscc"
Jul 13 18:50:15.953835 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:15.952602    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b7240a5a-c704-49a7-b152-bdeb36cee9d4-host\") pod \"node-ca-k2vg4\" (UID: \"b7240a5a-c704-49a7-b152-bdeb36cee9d4\") " pod="openshift-image-registry/node-ca-k2vg4"
Jul 13 18:50:15.953835 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:15.952606    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6ca2c2d9-346a-48c4-af31-ab3b2e66539f-cni-binary-copy\") pod \"multus-additional-cni-plugins-kgrbw\" (UID: \"6ca2c2d9-346a-48c4-af31-ab3b2e66539f\") " pod="openshift-multus/multus-additional-cni-plugins-kgrbw"
Jul 13 18:50:15.953835 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:15.952654    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6ca2c2d9-346a-48c4-af31-ab3b2e66539f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-kgrbw\" (UID: \"6ca2c2d9-346a-48c4-af31-ab3b2e66539f\") " pod="openshift-multus/multus-additional-cni-plugins-kgrbw"
Jul 13 18:50:15.953835 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:15.952684    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/34d8b129-0c9e-4cda-8669-d7595f3d01af-etc-selinux\") pod \"aws-ebs-csi-driver-node-psj7d\" (UID: \"34d8b129-0c9e-4cda-8669-d7595f3d01af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-psj7d"
Jul 13 18:50:15.953835 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:15.952746    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2f0655bd-4a09-4314-8c1b-e30191eaec0e-host-slash\") pod \"ovnkube-node-2bscc\" (UID: \"2f0655bd-4a09-4314-8c1b-e30191eaec0e\") " pod="openshift-ovn-kubernetes/ovnkube-node-2bscc"
Jul 13 18:50:15.953835 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:15.952792    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6ca2c2d9-346a-48c4-af31-ab3b2e66539f-cnibin\") pod \"multus-additional-cni-plugins-kgrbw\" (UID: \"6ca2c2d9-346a-48c4-af31-ab3b2e66539f\") " pod="openshift-multus/multus-additional-cni-plugins-kgrbw"
Jul 13 18:50:15.953835 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:15.952845    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ef958600-65d1-473a-b2dd-397c1dbbc8da-cni-binary-copy\") pod \"multus-qtpwz\" (UID: \"ef958600-65d1-473a-b2dd-397c1dbbc8da\") " pod="openshift-multus/multus-qtpwz"
Jul 13 18:50:15.953835 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:15.952862    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2f0655bd-4a09-4314-8c1b-e30191eaec0e-host-slash\") pod \"ovnkube-node-2bscc\" (UID: \"2f0655bd-4a09-4314-8c1b-e30191eaec0e\") " pod="openshift-ovn-kubernetes/ovnkube-node-2bscc"
Jul 13 18:50:15.953835 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:15.952877    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3a32830b-434d-4118-a91c-9dfdd29dcfa6-etc-sysctl-d\") pod \"tuned-gwvg9\" (UID: \"3a32830b-434d-4118-a91c-9dfdd29dcfa6\") " pod="openshift-cluster-node-tuning-operator/tuned-gwvg9"
Jul 13 18:50:15.953835 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:15.952929    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3a32830b-434d-4118-a91c-9dfdd29dcfa6-etc-systemd\") pod \"tuned-gwvg9\" (UID: \"3a32830b-434d-4118-a91c-9dfdd29dcfa6\") " pod="openshift-cluster-node-tuning-operator/tuned-gwvg9"
Jul 13 18:50:15.953835 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:15.952973    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3a32830b-434d-4118-a91c-9dfdd29dcfa6-tmp\") pod \"tuned-gwvg9\" (UID: \"3a32830b-434d-4118-a91c-9dfdd29dcfa6\") " pod="openshift-cluster-node-tuning-operator/tuned-gwvg9"
Jul 13 18:50:15.953835 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:15.953070    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2f0655bd-4a09-4314-8c1b-e30191eaec0e-host-run-netns\") pod \"ovnkube-node-2bscc\" (UID: \"2f0655bd-4a09-4314-8c1b-e30191eaec0e\") " pod="openshift-ovn-kubernetes/ovnkube-node-2bscc"
Jul 13 18:50:15.953835 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:15.953118    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2f0655bd-4a09-4314-8c1b-e30191eaec0e-host-cni-bin\") pod \"ovnkube-node-2bscc\" (UID: \"2f0655bd-4a09-4314-8c1b-e30191eaec0e\") " pod="openshift-ovn-kubernetes/ovnkube-node-2bscc"
Jul 13 18:50:15.953835 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:15.953184    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2f0655bd-4a09-4314-8c1b-e30191eaec0e-host-run-netns\") pod \"ovnkube-node-2bscc\" (UID: \"2f0655bd-4a09-4314-8c1b-e30191eaec0e\") " pod="openshift-ovn-kubernetes/ovnkube-node-2bscc"
Jul 13 18:50:15.954491 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:15.953235    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2f0655bd-4a09-4314-8c1b-e30191eaec0e-ovn-node-metrics-cert\") pod \"ovnkube-node-2bscc\" (UID: \"2f0655bd-4a09-4314-8c1b-e30191eaec0e\") " pod="openshift-ovn-kubernetes/ovnkube-node-2bscc"
Jul 13 18:50:15.954491 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:15.953269    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2f0655bd-4a09-4314-8c1b-e30191eaec0e-ovnkube-script-lib\") pod \"ovnkube-node-2bscc\" (UID: \"2f0655bd-4a09-4314-8c1b-e30191eaec0e\") " pod="openshift-ovn-kubernetes/ovnkube-node-2bscc"
Jul 13 18:50:15.954491 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:15.953300    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6ca2c2d9-346a-48c4-af31-ab3b2e66539f-system-cni-dir\") pod \"multus-additional-cni-plugins-kgrbw\" (UID: \"6ca2c2d9-346a-48c4-af31-ab3b2e66539f\") " pod="openshift-multus/multus-additional-cni-plugins-kgrbw"
Jul 13 18:50:15.954491 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:15.953336    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ef958600-65d1-473a-b2dd-397c1dbbc8da-host-var-lib-cni-bin\") pod \"multus-qtpwz\" (UID: \"ef958600-65d1-473a-b2dd-397c1dbbc8da\") " pod="openshift-multus/multus-qtpwz"
Jul 13 18:50:15.954491 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:15.953364    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6bbc0748-e539-4679-a100-1ba45794129b-dbus\") pod \"global-pull-secret-syncer-qlzr5\" (UID: \"6bbc0748-e539-4679-a100-1ba45794129b\") " pod="kube-system/global-pull-secret-syncer-qlzr5"
Jul 13 18:50:15.954491 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:15.953403    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6s6s\" (UniqueName: \"kubernetes.io/projected/d4fd4b59-2ecd-497b-a485-1e94101a9033-kube-api-access-m6s6s\") pod \"network-metrics-daemon-4sl67\" (UID: \"d4fd4b59-2ecd-497b-a485-1e94101a9033\") " pod="openshift-multus/network-metrics-daemon-4sl67"
Jul 13 18:50:15.954491 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:15.953434    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dz2kb\" (UniqueName: \"kubernetes.io/projected/10b9cbe5-c0a3-4d27-af48-2cebde3572bc-kube-api-access-dz2kb\") pod \"iptables-alerter-pxrrr\" (UID: \"10b9cbe5-c0a3-4d27-af48-2cebde3572bc\") " pod="openshift-network-operator/iptables-alerter-pxrrr"
Jul 13 18:50:15.954491 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:15.953462    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6bbc0748-e539-4679-a100-1ba45794129b-kubelet-config\") pod \"global-pull-secret-syncer-qlzr5\" (UID: \"6bbc0748-e539-4679-a100-1ba45794129b\") " pod="kube-system/global-pull-secret-syncer-qlzr5"
Jul 13 18:50:15.954491 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:15.953487    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/34d8b129-0c9e-4cda-8669-d7595f3d01af-socket-dir\") pod \"aws-ebs-csi-driver-node-psj7d\" (UID: \"34d8b129-0c9e-4cda-8669-d7595f3d01af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-psj7d"
Jul 13 18:50:15.954491 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:15.953513    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/34d8b129-0c9e-4cda-8669-d7595f3d01af-registration-dir\") pod \"aws-ebs-csi-driver-node-psj7d\" (UID: \"34d8b129-0c9e-4cda-8669-d7595f3d01af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-psj7d"
Jul 13 18:50:15.954491 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:15.953539    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3a32830b-434d-4118-a91c-9dfdd29dcfa6-etc-sysctl-conf\") pod \"tuned-gwvg9\" (UID: \"3a32830b-434d-4118-a91c-9dfdd29dcfa6\") " pod="openshift-cluster-node-tuning-operator/tuned-gwvg9"
Jul 13 18:50:15.954491 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:15.953564    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3a32830b-434d-4118-a91c-9dfdd29dcfa6-var-lib-kubelet\") pod \"tuned-gwvg9\" (UID: \"3a32830b-434d-4118-a91c-9dfdd29dcfa6\") " pod="openshift-cluster-node-tuning-operator/tuned-gwvg9"
Jul 13 18:50:15.954491 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:15.953593    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9dccc391-865f-45a2-9004-85e3830f39be-agent-certs\") pod \"konnectivity-agent-n4lpd\" (UID: \"9dccc391-865f-45a2-9004-85e3830f39be\") " pod="kube-system/konnectivity-agent-n4lpd"
Jul 13 18:50:15.954491 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:15.953620    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/34d8b129-0c9e-4cda-8669-d7595f3d01af-kubelet-dir\") pod \"aws-ebs-csi-driver-node-psj7d\" (UID: \"34d8b129-0c9e-4cda-8669-d7595f3d01af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-psj7d"
Jul 13 18:50:15.954491 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:15.953644    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/34d8b129-0c9e-4cda-8669-d7595f3d01af-sys-fs\") pod \"aws-ebs-csi-driver-node-psj7d\" (UID: \"34d8b129-0c9e-4cda-8669-d7595f3d01af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-psj7d"
Jul 13 18:50:15.954491 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:15.953667    2565 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 13 18:50:15.954491 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:15.953239    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2f0655bd-4a09-4314-8c1b-e30191eaec0e-host-cni-bin\") pod \"ovnkube-node-2bscc\" (UID: \"2f0655bd-4a09-4314-8c1b-e30191eaec0e\") " pod="openshift-ovn-kubernetes/ovnkube-node-2bscc"
Jul 13 18:50:15.955328 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:15.955198    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2f0655bd-4a09-4314-8c1b-e30191eaec0e-ovnkube-script-lib\") pod \"ovnkube-node-2bscc\" (UID: \"2f0655bd-4a09-4314-8c1b-e30191eaec0e\") " pod="openshift-ovn-kubernetes/ovnkube-node-2bscc"
Jul 13 18:50:15.957714 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:15.957690    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2f0655bd-4a09-4314-8c1b-e30191eaec0e-ovn-node-metrics-cert\") pod \"ovnkube-node-2bscc\" (UID: \"2f0655bd-4a09-4314-8c1b-e30191eaec0e\") " pod="openshift-ovn-kubernetes/ovnkube-node-2bscc"
Jul 13 18:50:15.957866 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:15.957826    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9dccc391-865f-45a2-9004-85e3830f39be-agent-certs\") pod \"konnectivity-agent-n4lpd\" (UID: \"9dccc391-865f-45a2-9004-85e3830f39be\") " pod="kube-system/konnectivity-agent-n4lpd"
Jul 13 18:50:15.966409 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:15.966369    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dz2kb\" (UniqueName: \"kubernetes.io/projected/10b9cbe5-c0a3-4d27-af48-2cebde3572bc-kube-api-access-dz2kb\") pod \"iptables-alerter-pxrrr\" (UID: \"10b9cbe5-c0a3-4d27-af48-2cebde3572bc\") " pod="openshift-network-operator/iptables-alerter-pxrrr"
Jul 13 18:50:15.966409 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:15.966388    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6s6s\" (UniqueName: \"kubernetes.io/projected/d4fd4b59-2ecd-497b-a485-1e94101a9033-kube-api-access-m6s6s\") pod \"network-metrics-daemon-4sl67\" (UID: \"d4fd4b59-2ecd-497b-a485-1e94101a9033\") " pod="openshift-multus/network-metrics-daemon-4sl67"
Jul 13 18:50:15.966806 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:15.966771    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmd5c\" (UniqueName: \"kubernetes.io/projected/6ae58b1e-3aaf-45dc-a741-2fb895468ca7-kube-api-access-rmd5c\") pod \"node-resolver-zcq5p\" (UID: \"6ae58b1e-3aaf-45dc-a741-2fb895468ca7\") " pod="openshift-dns/node-resolver-zcq5p"
Jul 13 18:50:15.966935 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:15.966880    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlt72\" (UniqueName: \"kubernetes.io/projected/b7240a5a-c704-49a7-b152-bdeb36cee9d4-kube-api-access-dlt72\") pod \"node-ca-k2vg4\" (UID: \"b7240a5a-c704-49a7-b152-bdeb36cee9d4\") " pod="openshift-image-registry/node-ca-k2vg4"
Jul 13 18:50:15.967158 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:15.967133    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdknc\" (UniqueName: \"kubernetes.io/projected/2f0655bd-4a09-4314-8c1b-e30191eaec0e-kube-api-access-xdknc\") pod \"ovnkube-node-2bscc\" (UID: \"2f0655bd-4a09-4314-8c1b-e30191eaec0e\") " pod="openshift-ovn-kubernetes/ovnkube-node-2bscc"
Jul 13 18:50:16.048342 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:16.048265    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-215.ec2.internal" event={"ID":"fce2836fb78b250794cad725b074cef4","Type":"ContainerStarted","Data":"69756716edb28839a1ab069baa44914aacb1d9a6b227a7b481fb74cd9d0e31db"}
Jul 13 18:50:16.054571 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:16.054535    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ef958600-65d1-473a-b2dd-397c1dbbc8da-os-release\") pod \"multus-qtpwz\" (UID: \"ef958600-65d1-473a-b2dd-397c1dbbc8da\") " pod="openshift-multus/multus-qtpwz"
Jul 13 18:50:16.054571 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:16.054575    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ef958600-65d1-473a-b2dd-397c1dbbc8da-host-var-lib-kubelet\") pod \"multus-qtpwz\" (UID: \"ef958600-65d1-473a-b2dd-397c1dbbc8da\") " pod="openshift-multus/multus-qtpwz"
Jul 13 18:50:16.054810 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:16.054596    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/34d8b129-0c9e-4cda-8669-d7595f3d01af-device-dir\") pod \"aws-ebs-csi-driver-node-psj7d\" (UID: \"34d8b129-0c9e-4cda-8669-d7595f3d01af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-psj7d"
Jul 13 18:50:16.054810 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:16.054615    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ef958600-65d1-473a-b2dd-397c1dbbc8da-host-run-netns\") pod \"multus-qtpwz\" (UID: \"ef958600-65d1-473a-b2dd-397c1dbbc8da\") " pod="openshift-multus/multus-qtpwz"
Jul 13 18:50:16.054810 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:16.054639    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ef958600-65d1-473a-b2dd-397c1dbbc8da-host-var-lib-cni-multus\") pod \"multus-qtpwz\" (UID: \"ef958600-65d1-473a-b2dd-397c1dbbc8da\") " pod="openshift-multus/multus-qtpwz"
Jul 13 18:50:16.054810 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:16.054703    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/34d8b129-0c9e-4cda-8669-d7595f3d01af-device-dir\") pod \"aws-ebs-csi-driver-node-psj7d\" (UID: \"34d8b129-0c9e-4cda-8669-d7595f3d01af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-psj7d"
Jul 13 18:50:16.054810 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:16.054701    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ef958600-65d1-473a-b2dd-397c1dbbc8da-os-release\") pod \"multus-qtpwz\" (UID: \"ef958600-65d1-473a-b2dd-397c1dbbc8da\") " pod="openshift-multus/multus-qtpwz"
Jul 13 18:50:16.054810 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:16.054722    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ef958600-65d1-473a-b2dd-397c1dbbc8da-host-var-lib-kubelet\") pod \"multus-qtpwz\" (UID: \"ef958600-65d1-473a-b2dd-397c1dbbc8da\") " pod="openshift-multus/multus-qtpwz"
Jul 13 18:50:16.054810 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:16.054782    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ef958600-65d1-473a-b2dd-397c1dbbc8da-multus-conf-dir\") pod \"multus-qtpwz\" (UID: \"ef958600-65d1-473a-b2dd-397c1dbbc8da\") " pod="openshift-multus/multus-qtpwz"
Jul 13 18:50:16.055144 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:16.054820    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ef958600-65d1-473a-b2dd-397c1dbbc8da-host-run-netns\") pod \"multus-qtpwz\" (UID: \"ef958600-65d1-473a-b2dd-397c1dbbc8da\") " pod="openshift-multus/multus-qtpwz"
Jul 13 18:50:16.055144 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:16.054838    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ef958600-65d1-473a-b2dd-397c1dbbc8da-etc-kubernetes\") pod \"multus-qtpwz\" (UID: \"ef958600-65d1-473a-b2dd-397c1dbbc8da\") " pod="openshift-multus/multus-qtpwz"
Jul 13 18:50:16.055144 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:16.054862    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ef958600-65d1-473a-b2dd-397c1dbbc8da-host-var-lib-cni-multus\") pod \"multus-qtpwz\" (UID: \"ef958600-65d1-473a-b2dd-397c1dbbc8da\") " pod="openshift-multus/multus-qtpwz"
Jul 13 18:50:16.055144 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:16.054865    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3a32830b-434d-4118-a91c-9dfdd29dcfa6-etc-tuned\") pod \"tuned-gwvg9\" (UID: \"3a32830b-434d-4118-a91c-9dfdd29dcfa6\") " pod="openshift-cluster-node-tuning-operator/tuned-gwvg9"
Jul 13 18:50:16.055144 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:16.054888    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvvhf\" (UniqueName: \"kubernetes.io/projected/76869c1f-1537-4e3e-b7e2-654a08cd2376-kube-api-access-qvvhf\") pod \"network-check-target-5tgm8\" (UID: \"76869c1f-1537-4e3e-b7e2-654a08cd2376\") " pod="openshift-network-diagnostics/network-check-target-5tgm8"
Jul 13 18:50:16.055144 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:16.054906    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3a32830b-434d-4118-a91c-9dfdd29dcfa6-etc-sysconfig\") pod \"tuned-gwvg9\" (UID: \"3a32830b-434d-4118-a91c-9dfdd29dcfa6\") " pod="openshift-cluster-node-tuning-operator/tuned-gwvg9"
Jul 13 18:50:16.055144 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:16.054924    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6ca2c2d9-346a-48c4-af31-ab3b2e66539f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-kgrbw\" (UID: \"6ca2c2d9-346a-48c4-af31-ab3b2e66539f\") " pod="openshift-multus/multus-additional-cni-plugins-kgrbw"
Jul 13 18:50:16.055144 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:16.054939    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ef958600-65d1-473a-b2dd-397c1dbbc8da-multus-conf-dir\") pod \"multus-qtpwz\" (UID: \"ef958600-65d1-473a-b2dd-397c1dbbc8da\") " pod="openshift-multus/multus-qtpwz"
Jul 13 18:50:16.055144 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:16.054960    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ef958600-65d1-473a-b2dd-397c1dbbc8da-etc-kubernetes\") pod \"multus-qtpwz\" (UID: \"ef958600-65d1-473a-b2dd-397c1dbbc8da\") " pod="openshift-multus/multus-qtpwz"
Jul 13 18:50:16.055144 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:16.054990    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3a32830b-434d-4118-a91c-9dfdd29dcfa6-etc-sysconfig\") pod \"tuned-gwvg9\" (UID: \"3a32830b-434d-4118-a91c-9dfdd29dcfa6\") " pod="openshift-cluster-node-tuning-operator/tuned-gwvg9"
Jul 13 18:50:16.055144 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:16.055032    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3a32830b-434d-4118-a91c-9dfdd29dcfa6-host\") pod \"tuned-gwvg9\" (UID: \"3a32830b-434d-4118-a91c-9dfdd29dcfa6\") " pod="openshift-cluster-node-tuning-operator/tuned-gwvg9"
Jul 13 18:50:16.055144 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:16.055081    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kx558\" (UniqueName: \"kubernetes.io/projected/6ca2c2d9-346a-48c4-af31-ab3b2e66539f-kube-api-access-kx558\") pod \"multus-additional-cni-plugins-kgrbw\" (UID: \"6ca2c2d9-346a-48c4-af31-ab3b2e66539f\") " pod="openshift-multus/multus-additional-cni-plugins-kgrbw"
Jul 13 18:50:16.055144 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:16.055100    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3a32830b-434d-4118-a91c-9dfdd29dcfa6-host\") pod \"tuned-gwvg9\" (UID: \"3a32830b-434d-4118-a91c-9dfdd29dcfa6\") " pod="openshift-cluster-node-tuning-operator/tuned-gwvg9"
Jul 13 18:50:16.055144 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:16.055109    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ef958600-65d1-473a-b2dd-397c1dbbc8da-multus-cni-dir\") pod \"multus-qtpwz\" (UID: \"ef958600-65d1-473a-b2dd-397c1dbbc8da\") " pod="openshift-multus/multus-qtpwz"
Jul 13 18:50:16.055144 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:16.055145    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ef958600-65d1-473a-b2dd-397c1dbbc8da-multus-daemon-config\") pod \"multus-qtpwz\" (UID: \"ef958600-65d1-473a-b2dd-397c1dbbc8da\") " pod="openshift-multus/multus-qtpwz"
Jul 13 18:50:16.055922 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:16.055167    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pz4z9\" (UniqueName: \"kubernetes.io/projected/34d8b129-0c9e-4cda-8669-d7595f3d01af-kube-api-access-pz4z9\") pod \"aws-ebs-csi-driver-node-psj7d\" (UID: \"34d8b129-0c9e-4cda-8669-d7595f3d01af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-psj7d"
Jul 13 18:50:16.055922 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:16.055169    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ef958600-65d1-473a-b2dd-397c1dbbc8da-multus-cni-dir\") pod \"multus-qtpwz\" (UID: \"ef958600-65d1-473a-b2dd-397c1dbbc8da\") " pod="openshift-multus/multus-qtpwz"
Jul 13 18:50:16.055922 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:16.055264    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6ca2c2d9-346a-48c4-af31-ab3b2e66539f-cni-binary-copy\") pod \"multus-additional-cni-plugins-kgrbw\" (UID: \"6ca2c2d9-346a-48c4-af31-ab3b2e66539f\") " pod="openshift-multus/multus-additional-cni-plugins-kgrbw"
Jul 13 18:50:16.055922 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:16.055301    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6ca2c2d9-346a-48c4-af31-ab3b2e66539f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-kgrbw\" (UID: \"6ca2c2d9-346a-48c4-af31-ab3b2e66539f\") " pod="openshift-multus/multus-additional-cni-plugins-kgrbw"
Jul 13 18:50:16.055922 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:16.055332    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/34d8b129-0c9e-4cda-8669-d7595f3d01af-etc-selinux\") pod \"aws-ebs-csi-driver-node-psj7d\" (UID: \"34d8b129-0c9e-4cda-8669-d7595f3d01af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-psj7d"
Jul 13 18:50:16.055922 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:16.055364    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6ca2c2d9-346a-48c4-af31-ab3b2e66539f-cnibin\") pod \"multus-additional-cni-plugins-kgrbw\" (UID: \"6ca2c2d9-346a-48c4-af31-ab3b2e66539f\") " pod="openshift-multus/multus-additional-cni-plugins-kgrbw"
Jul 13 18:50:16.055922 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:16.055394    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ef958600-65d1-473a-b2dd-397c1dbbc8da-cni-binary-copy\") pod \"multus-qtpwz\" (UID: \"ef958600-65d1-473a-b2dd-397c1dbbc8da\") " pod="openshift-multus/multus-qtpwz"
Jul 13 18:50:16.055922 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:16.055432    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3a32830b-434d-4118-a91c-9dfdd29dcfa6-etc-sysctl-d\") pod \"tuned-gwvg9\" (UID: \"3a32830b-434d-4118-a91c-9dfdd29dcfa6\") " pod="openshift-cluster-node-tuning-operator/tuned-gwvg9"
Jul 13 18:50:16.055922 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:16.055459    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3a32830b-434d-4118-a91c-9dfdd29dcfa6-etc-systemd\") pod \"tuned-gwvg9\" (UID: \"3a32830b-434d-4118-a91c-9dfdd29dcfa6\") " pod="openshift-cluster-node-tuning-operator/tuned-gwvg9"
Jul 13 18:50:16.055922 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:16.055461    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6ca2c2d9-346a-48c4-af31-ab3b2e66539f-cnibin\") pod \"multus-additional-cni-plugins-kgrbw\" (UID: \"6ca2c2d9-346a-48c4-af31-ab3b2e66539f\") " pod="openshift-multus/multus-additional-cni-plugins-kgrbw"
Jul 13 18:50:16.055922 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:16.055475    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/34d8b129-0c9e-4cda-8669-d7595f3d01af-etc-selinux\") pod \"aws-ebs-csi-driver-node-psj7d\" (UID: \"34d8b129-0c9e-4cda-8669-d7595f3d01af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-psj7d"
Jul 13 18:50:16.055922 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:16.055484    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3a32830b-434d-4118-a91c-9dfdd29dcfa6-tmp\") pod \"tuned-gwvg9\" (UID: \"3a32830b-434d-4118-a91c-9dfdd29dcfa6\") " pod="openshift-cluster-node-tuning-operator/tuned-gwvg9"
Jul 13 18:50:16.055922 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:16.055522    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6ca2c2d9-346a-48c4-af31-ab3b2e66539f-system-cni-dir\") pod \"multus-additional-cni-plugins-kgrbw\" (UID: \"6ca2c2d9-346a-48c4-af31-ab3b2e66539f\") " pod="openshift-multus/multus-additional-cni-plugins-kgrbw"
Jul 13 18:50:16.055922 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:16.055551    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ef958600-65d1-473a-b2dd-397c1dbbc8da-host-var-lib-cni-bin\") pod \"multus-qtpwz\" (UID: \"ef958600-65d1-473a-b2dd-397c1dbbc8da\") " pod="openshift-multus/multus-qtpwz"
Jul 13 18:50:16.055922 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:16.055575    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6bbc0748-e539-4679-a100-1ba45794129b-dbus\") pod \"global-pull-secret-syncer-qlzr5\" (UID: \"6bbc0748-e539-4679-a100-1ba45794129b\") " pod="kube-system/global-pull-secret-syncer-qlzr5"
Jul 13 18:50:16.055922 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:16.055587    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6ca2c2d9-346a-48c4-af31-ab3b2e66539f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-kgrbw\" (UID: \"6ca2c2d9-346a-48c4-af31-ab3b2e66539f\") " pod="openshift-multus/multus-additional-cni-plugins-kgrbw"
Jul 13 18:50:16.055922 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:16.055600    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6bbc0748-e539-4679-a100-1ba45794129b-kubelet-config\") pod \"global-pull-secret-syncer-qlzr5\" (UID: \"6bbc0748-e539-4679-a100-1ba45794129b\") " pod="kube-system/global-pull-secret-syncer-qlzr5"
Jul 13 18:50:16.056976 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:16.055629    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/34d8b129-0c9e-4cda-8669-d7595f3d01af-socket-dir\") pod \"aws-ebs-csi-driver-node-psj7d\" (UID: \"34d8b129-0c9e-4cda-8669-d7595f3d01af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-psj7d"
Jul 13 18:50:16.056976 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:16.055652    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ef958600-65d1-473a-b2dd-397c1dbbc8da-host-var-lib-cni-bin\") pod \"multus-qtpwz\" (UID: \"ef958600-65d1-473a-b2dd-397c1dbbc8da\") " pod="openshift-multus/multus-qtpwz"
Jul 13 18:50:16.056976 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:16.055659    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/34d8b129-0c9e-4cda-8669-d7595f3d01af-registration-dir\") pod \"aws-ebs-csi-driver-node-psj7d\" (UID: \"34d8b129-0c9e-4cda-8669-d7595f3d01af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-psj7d"
Jul 13 18:50:16.056976 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:16.055699    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6ca2c2d9-346a-48c4-af31-ab3b2e66539f-system-cni-dir\") pod \"multus-additional-cni-plugins-kgrbw\" (UID: \"6ca2c2d9-346a-48c4-af31-ab3b2e66539f\") " pod="openshift-multus/multus-additional-cni-plugins-kgrbw"
Jul 13 18:50:16.056976 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:16.055750    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/34d8b129-0c9e-4cda-8669-d7595f3d01af-registration-dir\") pod \"aws-ebs-csi-driver-node-psj7d\" (UID: \"34d8b129-0c9e-4cda-8669-d7595f3d01af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-psj7d"
Jul 13 18:50:16.056976 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:16.055802    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3a32830b-434d-4118-a91c-9dfdd29dcfa6-etc-sysctl-conf\") pod \"tuned-gwvg9\" (UID: \"3a32830b-434d-4118-a91c-9dfdd29dcfa6\") " pod="openshift-cluster-node-tuning-operator/tuned-gwvg9"
Jul 13 18:50:16.056976 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:16.055805    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6bbc0748-e539-4679-a100-1ba45794129b-dbus\") pod \"global-pull-secret-syncer-qlzr5\" (UID: \"6bbc0748-e539-4679-a100-1ba45794129b\") " pod="kube-system/global-pull-secret-syncer-qlzr5"
Jul 13 18:50:16.056976 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:16.055814    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ef958600-65d1-473a-b2dd-397c1dbbc8da-multus-daemon-config\") pod \"multus-qtpwz\" (UID: \"ef958600-65d1-473a-b2dd-397c1dbbc8da\") " pod="openshift-multus/multus-qtpwz"
Jul 13 18:50:16.056976 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:16.055854    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3a32830b-434d-4118-a91c-9dfdd29dcfa6-var-lib-kubelet\") pod \"tuned-gwvg9\" (UID: \"3a32830b-434d-4118-a91c-9dfdd29dcfa6\") " pod="openshift-cluster-node-tuning-operator/tuned-gwvg9"
Jul 13 18:50:16.056976 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:16.055875    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6bbc0748-e539-4679-a100-1ba45794129b-kubelet-config\") pod \"global-pull-secret-syncer-qlzr5\" (UID: \"6bbc0748-e539-4679-a100-1ba45794129b\") " pod="kube-system/global-pull-secret-syncer-qlzr5"
Jul 13 18:50:16.056976 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:16.055883    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/34d8b129-0c9e-4cda-8669-d7595f3d01af-kubelet-dir\") pod \"aws-ebs-csi-driver-node-psj7d\" (UID: \"34d8b129-0c9e-4cda-8669-d7595f3d01af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-psj7d"
Jul 13 18:50:16.056976 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:16.055907    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/34d8b129-0c9e-4cda-8669-d7595f3d01af-sys-fs\") pod \"aws-ebs-csi-driver-node-psj7d\" (UID: \"34d8b129-0c9e-4cda-8669-d7595f3d01af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-psj7d"
Jul 13 18:50:16.056976 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:16.055929    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/34d8b129-0c9e-4cda-8669-d7595f3d01af-socket-dir\") pod \"aws-ebs-csi-driver-node-psj7d\" (UID: \"34d8b129-0c9e-4cda-8669-d7595f3d01af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-psj7d"
Jul 13 18:50:16.056976 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:16.055926    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6ca2c2d9-346a-48c4-af31-ab3b2e66539f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-kgrbw\" (UID: \"6ca2c2d9-346a-48c4-af31-ab3b2e66539f\") " pod="openshift-multus/multus-additional-cni-plugins-kgrbw"
Jul 13 18:50:16.056976 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:16.055955    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ef958600-65d1-473a-b2dd-397c1dbbc8da-system-cni-dir\") pod \"multus-qtpwz\" (UID: \"ef958600-65d1-473a-b2dd-397c1dbbc8da\") " pod="openshift-multus/multus-qtpwz"
Jul 13 18:50:16.056976 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:16.055984    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3a32830b-434d-4118-a91c-9dfdd29dcfa6-var-lib-kubelet\") pod \"tuned-gwvg9\" (UID: \"3a32830b-434d-4118-a91c-9dfdd29dcfa6\") " pod="openshift-cluster-node-tuning-operator/tuned-gwvg9"
Jul 13 18:50:16.056976 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:16.055995    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ef958600-65d1-473a-b2dd-397c1dbbc8da-system-cni-dir\") pod \"multus-qtpwz\" (UID: \"ef958600-65d1-473a-b2dd-397c1dbbc8da\") " pod="openshift-multus/multus-qtpwz"
Jul 13 18:50:16.057822 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:16.055989    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ef958600-65d1-473a-b2dd-397c1dbbc8da-host-run-k8s-cni-cncf-io\") pod \"multus-qtpwz\" (UID: \"ef958600-65d1-473a-b2dd-397c1dbbc8da\") " pod="openshift-multus/multus-qtpwz"
Jul 13 18:50:16.057822 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:16.056031    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3a32830b-434d-4118-a91c-9dfdd29dcfa6-etc-sysctl-conf\") pod \"tuned-gwvg9\" (UID: \"3a32830b-434d-4118-a91c-9dfdd29dcfa6\") " pod="openshift-cluster-node-tuning-operator/tuned-gwvg9"
Jul 13 18:50:16.057822 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:16.056036    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ef958600-65d1-473a-b2dd-397c1dbbc8da-hostroot\") pod \"multus-qtpwz\" (UID: \"ef958600-65d1-473a-b2dd-397c1dbbc8da\") " pod="openshift-multus/multus-qtpwz"
Jul 13 18:50:16.057822 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:16.056042    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ef958600-65d1-473a-b2dd-397c1dbbc8da-cni-binary-copy\") pod \"multus-qtpwz\" (UID: \"ef958600-65d1-473a-b2dd-397c1dbbc8da\") " pod="openshift-multus/multus-qtpwz"
Jul 13 18:50:16.057822 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:16.056024    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ef958600-65d1-473a-b2dd-397c1dbbc8da-host-run-k8s-cni-cncf-io\") pod \"multus-qtpwz\" (UID: \"ef958600-65d1-473a-b2dd-397c1dbbc8da\") " pod="openshift-multus/multus-qtpwz"
Jul 13 18:50:16.057822 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:16.056052    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6ca2c2d9-346a-48c4-af31-ab3b2e66539f-cni-binary-copy\") pod \"multus-additional-cni-plugins-kgrbw\" (UID: \"6ca2c2d9-346a-48c4-af31-ab3b2e66539f\") " pod="openshift-multus/multus-additional-cni-plugins-kgrbw"
Jul 13 18:50:16.057822 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:16.056042    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/34d8b129-0c9e-4cda-8669-d7595f3d01af-kubelet-dir\") pod \"aws-ebs-csi-driver-node-psj7d\" (UID: \"34d8b129-0c9e-4cda-8669-d7595f3d01af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-psj7d"
Jul 13 18:50:16.057822 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:16.056071    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ef958600-65d1-473a-b2dd-397c1dbbc8da-cnibin\") pod \"multus-qtpwz\" (UID: \"ef958600-65d1-473a-b2dd-397c1dbbc8da\") " pod="openshift-multus/multus-qtpwz"
Jul 13 18:50:16.057822 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:16.056090    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/34d8b129-0c9e-4cda-8669-d7595f3d01af-sys-fs\") pod \"aws-ebs-csi-driver-node-psj7d\" (UID: \"34d8b129-0c9e-4cda-8669-d7595f3d01af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-psj7d"
Jul 13 18:50:16.057822 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:16.056094    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3a32830b-434d-4118-a91c-9dfdd29dcfa6-etc-systemd\") pod \"tuned-gwvg9\" (UID: \"3a32830b-434d-4118-a91c-9dfdd29dcfa6\") " pod="openshift-cluster-node-tuning-operator/tuned-gwvg9"
Jul 13 18:50:16.057822 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:16.056097    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dxwr\" (UniqueName: \"kubernetes.io/projected/ef958600-65d1-473a-b2dd-397c1dbbc8da-kube-api-access-9dxwr\") pod \"multus-qtpwz\" (UID: \"ef958600-65d1-473a-b2dd-397c1dbbc8da\") " pod="openshift-multus/multus-qtpwz"
Jul 13 18:50:16.057822 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:16.056133    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6bbc0748-e539-4679-a100-1ba45794129b-original-pull-secret\") pod \"global-pull-secret-syncer-qlzr5\" (UID: \"6bbc0748-e539-4679-a100-1ba45794129b\") " pod="kube-system/global-pull-secret-syncer-qlzr5"
Jul 13 18:50:16.057822 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:16.056139    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ef958600-65d1-473a-b2dd-397c1dbbc8da-cnibin\") pod \"multus-qtpwz\" (UID: \"ef958600-65d1-473a-b2dd-397c1dbbc8da\") " pod="openshift-multus/multus-qtpwz"
Jul 13 18:50:16.057822 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:16.056108    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ef958600-65d1-473a-b2dd-397c1dbbc8da-hostroot\") pod \"multus-qtpwz\" (UID: \"ef958600-65d1-473a-b2dd-397c1dbbc8da\") " pod="openshift-multus/multus-qtpwz"
Jul 13 18:50:16.057822 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:16.056158    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3a32830b-434d-4118-a91c-9dfdd29dcfa6-sys\") pod \"tuned-gwvg9\" (UID: \"3a32830b-434d-4118-a91c-9dfdd29dcfa6\") " pod="openshift-cluster-node-tuning-operator/tuned-gwvg9"
Jul 13 18:50:16.057822 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:16.056184    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3a32830b-434d-4118-a91c-9dfdd29dcfa6-etc-sysctl-d\") pod \"tuned-gwvg9\" (UID: \"3a32830b-434d-4118-a91c-9dfdd29dcfa6\") " pod="openshift-cluster-node-tuning-operator/tuned-gwvg9"
Jul 13 18:50:16.057822 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:16.056194    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6ca2c2d9-346a-48c4-af31-ab3b2e66539f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-kgrbw\" (UID: \"6ca2c2d9-346a-48c4-af31-ab3b2e66539f\") " pod="openshift-multus/multus-additional-cni-plugins-kgrbw"
Jul 13 18:50:16.057822 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:16.056199    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3a32830b-434d-4118-a91c-9dfdd29dcfa6-sys\") pod \"tuned-gwvg9\" (UID: \"3a32830b-434d-4118-a91c-9dfdd29dcfa6\") " pod="openshift-cluster-node-tuning-operator/tuned-gwvg9"
Jul 13 18:50:16.058708 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:16.056220    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ef958600-65d1-473a-b2dd-397c1dbbc8da-host-run-multus-certs\") pod \"multus-qtpwz\" (UID: \"ef958600-65d1-473a-b2dd-397c1dbbc8da\") " pod="openshift-multus/multus-qtpwz"
Jul 13 18:50:16.058708 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:50:16.056231    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 13 18:50:16.058708 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:16.056247    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3a32830b-434d-4118-a91c-9dfdd29dcfa6-etc-modprobe-d\") pod \"tuned-gwvg9\" (UID: \"3a32830b-434d-4118-a91c-9dfdd29dcfa6\") " pod="openshift-cluster-node-tuning-operator/tuned-gwvg9"
Jul 13 18:50:16.058708 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:16.056275    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ef958600-65d1-473a-b2dd-397c1dbbc8da-multus-socket-dir-parent\") pod \"multus-qtpwz\" (UID: \"ef958600-65d1-473a-b2dd-397c1dbbc8da\") " pod="openshift-multus/multus-qtpwz"
Jul 13 18:50:16.058708 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:50:16.056290    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6bbc0748-e539-4679-a100-1ba45794129b-original-pull-secret podName:6bbc0748-e539-4679-a100-1ba45794129b nodeName:}" failed. No retries permitted until 2026-07-13 18:50:16.55627678 +0000 UTC m=+3.081410176 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6bbc0748-e539-4679-a100-1ba45794129b-original-pull-secret") pod "global-pull-secret-syncer-qlzr5" (UID: "6bbc0748-e539-4679-a100-1ba45794129b") : object "kube-system"/"original-pull-secret" not registered
Jul 13 18:50:16.058708 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:16.056312    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3a32830b-434d-4118-a91c-9dfdd29dcfa6-etc-kubernetes\") pod \"tuned-gwvg9\" (UID: \"3a32830b-434d-4118-a91c-9dfdd29dcfa6\") " pod="openshift-cluster-node-tuning-operator/tuned-gwvg9"
Jul 13 18:50:16.058708 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:16.056330    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3a32830b-434d-4118-a91c-9dfdd29dcfa6-run\") pod \"tuned-gwvg9\" (UID: \"3a32830b-434d-4118-a91c-9dfdd29dcfa6\") " pod="openshift-cluster-node-tuning-operator/tuned-gwvg9"
Jul 13 18:50:16.058708 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:16.056355    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xbqs\" (UniqueName: \"kubernetes.io/projected/3a32830b-434d-4118-a91c-9dfdd29dcfa6-kube-api-access-5xbqs\") pod \"tuned-gwvg9\" (UID: \"3a32830b-434d-4118-a91c-9dfdd29dcfa6\") " pod="openshift-cluster-node-tuning-operator/tuned-gwvg9"
Jul 13 18:50:16.058708 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:16.056385    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ef958600-65d1-473a-b2dd-397c1dbbc8da-multus-socket-dir-parent\") pod \"multus-qtpwz\" (UID: \"ef958600-65d1-473a-b2dd-397c1dbbc8da\") " pod="openshift-multus/multus-qtpwz"
Jul 13 18:50:16.058708 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:16.056396    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ef958600-65d1-473a-b2dd-397c1dbbc8da-host-run-multus-certs\") pod \"multus-qtpwz\" (UID: \"ef958600-65d1-473a-b2dd-397c1dbbc8da\") " pod="openshift-multus/multus-qtpwz"
Jul 13 18:50:16.058708 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:16.056398    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6ca2c2d9-346a-48c4-af31-ab3b2e66539f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-kgrbw\" (UID: \"6ca2c2d9-346a-48c4-af31-ab3b2e66539f\") " pod="openshift-multus/multus-additional-cni-plugins-kgrbw"
Jul 13 18:50:16.058708 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:16.056438    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3a32830b-434d-4118-a91c-9dfdd29dcfa6-etc-kubernetes\") pod \"tuned-gwvg9\" (UID: \"3a32830b-434d-4118-a91c-9dfdd29dcfa6\") " pod="openshift-cluster-node-tuning-operator/tuned-gwvg9"
Jul 13 18:50:16.058708 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:16.056438    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/6ca2c2d9-346a-48c4-af31-ab3b2e66539f-os-release\") pod \"multus-additional-cni-plugins-kgrbw\" (UID: \"6ca2c2d9-346a-48c4-af31-ab3b2e66539f\") " pod="openshift-multus/multus-additional-cni-plugins-kgrbw"
Jul 13 18:50:16.058708 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:16.056474    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3a32830b-434d-4118-a91c-9dfdd29dcfa6-lib-modules\") pod \"tuned-gwvg9\" (UID: \"3a32830b-434d-4118-a91c-9dfdd29dcfa6\") " pod="openshift-cluster-node-tuning-operator/tuned-gwvg9"
Jul 13 18:50:16.058708 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:16.056483    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3a32830b-434d-4118-a91c-9dfdd29dcfa6-etc-modprobe-d\") pod \"tuned-gwvg9\" (UID: \"3a32830b-434d-4118-a91c-9dfdd29dcfa6\") " pod="openshift-cluster-node-tuning-operator/tuned-gwvg9"
Jul 13 18:50:16.058708 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:16.056489    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/6ca2c2d9-346a-48c4-af31-ab3b2e66539f-os-release\") pod \"multus-additional-cni-plugins-kgrbw\" (UID: \"6ca2c2d9-346a-48c4-af31-ab3b2e66539f\") " pod="openshift-multus/multus-additional-cni-plugins-kgrbw"
Jul 13 18:50:16.058708 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:16.056477    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3a32830b-434d-4118-a91c-9dfdd29dcfa6-run\") pod \"tuned-gwvg9\" (UID: \"3a32830b-434d-4118-a91c-9dfdd29dcfa6\") " pod="openshift-cluster-node-tuning-operator/tuned-gwvg9"
Jul 13 18:50:16.059319 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:16.056583    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3a32830b-434d-4118-a91c-9dfdd29dcfa6-lib-modules\") pod \"tuned-gwvg9\" (UID: \"3a32830b-434d-4118-a91c-9dfdd29dcfa6\") " pod="openshift-cluster-node-tuning-operator/tuned-gwvg9"
Jul 13 18:50:16.059319 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:16.058709    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3a32830b-434d-4118-a91c-9dfdd29dcfa6-etc-tuned\") pod \"tuned-gwvg9\" (UID: \"3a32830b-434d-4118-a91c-9dfdd29dcfa6\") " pod="openshift-cluster-node-tuning-operator/tuned-gwvg9"
Jul 13 18:50:16.059319 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:16.058878    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3a32830b-434d-4118-a91c-9dfdd29dcfa6-tmp\") pod \"tuned-gwvg9\" (UID: \"3a32830b-434d-4118-a91c-9dfdd29dcfa6\") " pod="openshift-cluster-node-tuning-operator/tuned-gwvg9"
Jul 13 18:50:16.067324 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:50:16.067246    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 13 18:50:16.067324 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:50:16.067278    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 13 18:50:16.067324 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:50:16.067291    2565 projected.go:196] Error preparing data for projected volume kube-api-access-qvvhf for pod openshift-network-diagnostics/network-check-target-5tgm8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 13 18:50:16.067583 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:50:16.067372    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/76869c1f-1537-4e3e-b7e2-654a08cd2376-kube-api-access-qvvhf podName:76869c1f-1537-4e3e-b7e2-654a08cd2376 nodeName:}" failed. No retries permitted until 2026-07-13 18:50:16.567354206 +0000 UTC m=+3.092487606 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-qvvhf" (UniqueName: "kubernetes.io/projected/76869c1f-1537-4e3e-b7e2-654a08cd2376-kube-api-access-qvvhf") pod "network-check-target-5tgm8" (UID: "76869c1f-1537-4e3e-b7e2-654a08cd2376") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 13 18:50:16.068039 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:16.068018    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dxwr\" (UniqueName: \"kubernetes.io/projected/ef958600-65d1-473a-b2dd-397c1dbbc8da-kube-api-access-9dxwr\") pod \"multus-qtpwz\" (UID: \"ef958600-65d1-473a-b2dd-397c1dbbc8da\") " pod="openshift-multus/multus-qtpwz"
Jul 13 18:50:16.068111 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:16.068022    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kx558\" (UniqueName: \"kubernetes.io/projected/6ca2c2d9-346a-48c4-af31-ab3b2e66539f-kube-api-access-kx558\") pod \"multus-additional-cni-plugins-kgrbw\" (UID: \"6ca2c2d9-346a-48c4-af31-ab3b2e66539f\") " pod="openshift-multus/multus-additional-cni-plugins-kgrbw"
Jul 13 18:50:16.071992 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:16.071959    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xbqs\" (UniqueName: \"kubernetes.io/projected/3a32830b-434d-4118-a91c-9dfdd29dcfa6-kube-api-access-5xbqs\") pod \"tuned-gwvg9\" (UID: \"3a32830b-434d-4118-a91c-9dfdd29dcfa6\") " pod="openshift-cluster-node-tuning-operator/tuned-gwvg9"
Jul 13 18:50:16.073196 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:16.073165    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pz4z9\" (UniqueName: \"kubernetes.io/projected/34d8b129-0c9e-4cda-8669-d7595f3d01af-kube-api-access-pz4z9\") pod \"aws-ebs-csi-driver-node-psj7d\" (UID: \"34d8b129-0c9e-4cda-8669-d7595f3d01af\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-psj7d"
Jul 13 18:50:16.125673 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:16.125630    2565 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 13 18:50:16.142955 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:16.142919    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-n4lpd"
Jul 13 18:50:16.157155 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:16.157118    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-pxrrr"
Jul 13 18:50:16.169366 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:16.169334    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-2bscc"
Jul 13 18:50:16.177169 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:16.177141    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-k2vg4"
Jul 13 18:50:16.186280 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:16.186229    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-zcq5p"
Jul 13 18:50:16.194558 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:16.194524    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-kgrbw"
Jul 13 18:50:16.202633 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:16.202604    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-qtpwz"
Jul 13 18:50:16.211827 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:16.211796    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-psj7d"
Jul 13 18:50:16.219936 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:16.219906    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-gwvg9"
Jul 13 18:50:16.460454 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:16.460418    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d4fd4b59-2ecd-497b-a485-1e94101a9033-metrics-certs\") pod \"network-metrics-daemon-4sl67\" (UID: \"d4fd4b59-2ecd-497b-a485-1e94101a9033\") " pod="openshift-multus/network-metrics-daemon-4sl67"
Jul 13 18:50:16.460664 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:50:16.460601    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 13 18:50:16.460753 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:50:16.460676    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d4fd4b59-2ecd-497b-a485-1e94101a9033-metrics-certs podName:d4fd4b59-2ecd-497b-a485-1e94101a9033 nodeName:}" failed. No retries permitted until 2026-07-13 18:50:17.460658623 +0000 UTC m=+3.985792026 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d4fd4b59-2ecd-497b-a485-1e94101a9033-metrics-certs") pod "network-metrics-daemon-4sl67" (UID: "d4fd4b59-2ecd-497b-a485-1e94101a9033") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 13 18:50:16.561918 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:16.561860    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6bbc0748-e539-4679-a100-1ba45794129b-original-pull-secret\") pod \"global-pull-secret-syncer-qlzr5\" (UID: \"6bbc0748-e539-4679-a100-1ba45794129b\") " pod="kube-system/global-pull-secret-syncer-qlzr5"
Jul 13 18:50:16.562090 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:50:16.562050    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 13 18:50:16.562144 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:50:16.562132    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6bbc0748-e539-4679-a100-1ba45794129b-original-pull-secret podName:6bbc0748-e539-4679-a100-1ba45794129b nodeName:}" failed. No retries permitted until 2026-07-13 18:50:17.562113981 +0000 UTC m=+4.087247380 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6bbc0748-e539-4679-a100-1ba45794129b-original-pull-secret") pod "global-pull-secret-syncer-qlzr5" (UID: "6bbc0748-e539-4679-a100-1ba45794129b") : object "kube-system"/"original-pull-secret" not registered
Jul 13 18:50:16.662898 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:16.662851    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvvhf\" (UniqueName: \"kubernetes.io/projected/76869c1f-1537-4e3e-b7e2-654a08cd2376-kube-api-access-qvvhf\") pod \"network-check-target-5tgm8\" (UID: \"76869c1f-1537-4e3e-b7e2-654a08cd2376\") " pod="openshift-network-diagnostics/network-check-target-5tgm8"
Jul 13 18:50:16.663120 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:50:16.663029    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 13 18:50:16.663120 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:50:16.663045    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 13 18:50:16.663120 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:50:16.663055    2565 projected.go:196] Error preparing data for projected volume kube-api-access-qvvhf for pod openshift-network-diagnostics/network-check-target-5tgm8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 13 18:50:16.663120 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:50:16.663118    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/76869c1f-1537-4e3e-b7e2-654a08cd2376-kube-api-access-qvvhf podName:76869c1f-1537-4e3e-b7e2-654a08cd2376 nodeName:}" failed. No retries permitted until 2026-07-13 18:50:17.663103764 +0000 UTC m=+4.188237159 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-qvvhf" (UniqueName: "kubernetes.io/projected/76869c1f-1537-4e3e-b7e2-654a08cd2376-kube-api-access-qvvhf") pod "network-check-target-5tgm8" (UID: "76869c1f-1537-4e3e-b7e2-654a08cd2376") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 13 18:50:16.862867 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:16.862653    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef958600_65d1_473a_b2dd_397c1dbbc8da.slice/crio-a1fc85bc70202ce055124509a3701b96ea401ae0ec3ff243609fb3118258bca9 WatchSource:0}: Error finding container a1fc85bc70202ce055124509a3701b96ea401ae0ec3ff243609fb3118258bca9: Status 404 returned error can't find the container with id a1fc85bc70202ce055124509a3701b96ea401ae0ec3ff243609fb3118258bca9
Jul 13 18:50:16.869866 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:16.869836    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7240a5a_c704_49a7_b152_bdeb36cee9d4.slice/crio-2fed2446b40b2adeb1a2be6db1f6668efdce0cc65509c2a3839c5ea494a68f1a WatchSource:0}: Error finding container 2fed2446b40b2adeb1a2be6db1f6668efdce0cc65509c2a3839c5ea494a68f1a: Status 404 returned error can't find the container with id 2fed2446b40b2adeb1a2be6db1f6668efdce0cc65509c2a3839c5ea494a68f1a
Jul 13 18:50:16.870584 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:16.870560    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ae58b1e_3aaf_45dc_a741_2fb895468ca7.slice/crio-697c58e52e9cc57c5a9c2bf9fe1294b97f756f92f27b0f5b0b00388b30b1f79d WatchSource:0}: Error finding container 697c58e52e9cc57c5a9c2bf9fe1294b97f756f92f27b0f5b0b00388b30b1f79d: Status 404 returned error can't find the container with id 697c58e52e9cc57c5a9c2bf9fe1294b97f756f92f27b0f5b0b00388b30b1f79d
Jul 13 18:50:16.873057 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:16.873031    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34d8b129_0c9e_4cda_8669_d7595f3d01af.slice/crio-b067dc45adc068cd25c1923c8f7b9e7a6f0d50276c0e56adb0715f6eb934f0d4 WatchSource:0}: Error finding container b067dc45adc068cd25c1923c8f7b9e7a6f0d50276c0e56adb0715f6eb934f0d4: Status 404 returned error can't find the container with id b067dc45adc068cd25c1923c8f7b9e7a6f0d50276c0e56adb0715f6eb934f0d4
Jul 13 18:50:16.879163 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:16.875802    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ca2c2d9_346a_48c4_af31_ab3b2e66539f.slice/crio-285f618c65ca3659c9df73a5d92d2980620038620ec6b7f7779f4a6f9202f10c WatchSource:0}: Error finding container 285f618c65ca3659c9df73a5d92d2980620038620ec6b7f7779f4a6f9202f10c: Status 404 returned error can't find the container with id 285f618c65ca3659c9df73a5d92d2980620038620ec6b7f7779f4a6f9202f10c
Jul 13 18:50:16.879163 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:16.877750    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a32830b_434d_4118_a91c_9dfdd29dcfa6.slice/crio-cdeba09c0f87230468235e7b6ca6ec89874f49de5f04cd9f8c1a8417edd8665d WatchSource:0}: Error finding container cdeba09c0f87230468235e7b6ca6ec89874f49de5f04cd9f8c1a8417edd8665d: Status 404 returned error can't find the container with id cdeba09c0f87230468235e7b6ca6ec89874f49de5f04cd9f8c1a8417edd8665d
Jul 13 18:50:16.879163 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:16.878599    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10b9cbe5_c0a3_4d27_af48_2cebde3572bc.slice/crio-70ed85d21505b7b2d700f994bdcef2f133bc3e05d41f880a7538434f6603673c WatchSource:0}: Error finding container 70ed85d21505b7b2d700f994bdcef2f133bc3e05d41f880a7538434f6603673c: Status 404 returned error can't find the container with id 70ed85d21505b7b2d700f994bdcef2f133bc3e05d41f880a7538434f6603673c
Jul 13 18:50:16.879163 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:16.879048    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f0655bd_4a09_4314_8c1b_e30191eaec0e.slice/crio-344132a5ce400aeaa7db8cefcfd60eb36a50eff90afe57f4f09d42b67e17042d WatchSource:0}: Error finding container 344132a5ce400aeaa7db8cefcfd60eb36a50eff90afe57f4f09d42b67e17042d: Status 404 returned error can't find the container with id 344132a5ce400aeaa7db8cefcfd60eb36a50eff90afe57f4f09d42b67e17042d
Jul 13 18:50:16.891020 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:16.890999    2565 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 13 18:50:16.917917 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:16.917859    2565 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-12 18:45:16 +0000 UTC" deadline="2028-02-04 08:05:26.372674142 +0000 UTC"
Jul 13 18:50:16.917917 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:16.917910    2565 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13693h15m9.454766538s"
Jul 13 18:50:17.042678 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:17.042642    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qlzr5"
Jul 13 18:50:17.042856 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:50:17.042783    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qlzr5" podUID="6bbc0748-e539-4679-a100-1ba45794129b"
Jul 13 18:50:17.052606 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:17.052562    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-215.ec2.internal" event={"ID":"33d859ca02a99923eded717ac20e75f3","Type":"ContainerStarted","Data":"65e982d966a5cf7afa79ca2d5a5e1e57d01501cf5add9a12a5ed816b90c5ddbf"}
Jul 13 18:50:17.053778 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:17.053750    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-pxrrr" event={"ID":"10b9cbe5-c0a3-4d27-af48-2cebde3572bc","Type":"ContainerStarted","Data":"70ed85d21505b7b2d700f994bdcef2f133bc3e05d41f880a7538434f6603673c"}
Jul 13 18:50:17.054925 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:17.054900    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-gwvg9" event={"ID":"3a32830b-434d-4118-a91c-9dfdd29dcfa6","Type":"ContainerStarted","Data":"cdeba09c0f87230468235e7b6ca6ec89874f49de5f04cd9f8c1a8417edd8665d"}
Jul 13 18:50:17.055929 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:17.055908    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-zcq5p" event={"ID":"6ae58b1e-3aaf-45dc-a741-2fb895468ca7","Type":"ContainerStarted","Data":"697c58e52e9cc57c5a9c2bf9fe1294b97f756f92f27b0f5b0b00388b30b1f79d"}
Jul 13 18:50:17.057025 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:17.057003    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-k2vg4" event={"ID":"b7240a5a-c704-49a7-b152-bdeb36cee9d4","Type":"ContainerStarted","Data":"2fed2446b40b2adeb1a2be6db1f6668efdce0cc65509c2a3839c5ea494a68f1a"}
Jul 13 18:50:17.058162 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:17.058139    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-n4lpd" event={"ID":"9dccc391-865f-45a2-9004-85e3830f39be","Type":"ContainerStarted","Data":"a7f3fdb1bed77f80c64e8b4562991dcb4ca928c1ecb96c80a12d519a04bb4bda"}
Jul 13 18:50:17.059325 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:17.059304    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2bscc" event={"ID":"2f0655bd-4a09-4314-8c1b-e30191eaec0e","Type":"ContainerStarted","Data":"344132a5ce400aeaa7db8cefcfd60eb36a50eff90afe57f4f09d42b67e17042d"}
Jul 13 18:50:17.060411 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:17.060392    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kgrbw" event={"ID":"6ca2c2d9-346a-48c4-af31-ab3b2e66539f","Type":"ContainerStarted","Data":"285f618c65ca3659c9df73a5d92d2980620038620ec6b7f7779f4a6f9202f10c"}
Jul 13 18:50:17.061488 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:17.061468    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-psj7d" event={"ID":"34d8b129-0c9e-4cda-8669-d7595f3d01af","Type":"ContainerStarted","Data":"b067dc45adc068cd25c1923c8f7b9e7a6f0d50276c0e56adb0715f6eb934f0d4"}
Jul 13 18:50:17.062465 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:17.062445    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-qtpwz" event={"ID":"ef958600-65d1-473a-b2dd-397c1dbbc8da","Type":"ContainerStarted","Data":"a1fc85bc70202ce055124509a3701b96ea401ae0ec3ff243609fb3118258bca9"}
Jul 13 18:50:17.071018 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:17.070957    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-215.ec2.internal" podStartSLOduration=2.070940769 podStartE2EDuration="2.070940769s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 18:50:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:50:17.07038974 +0000 UTC m=+3.595523253" watchObservedRunningTime="2026-07-13 18:50:17.070940769 +0000 UTC m=+3.596074188"
Jul 13 18:50:17.114343 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:17.114241    2565 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 13 18:50:17.472101 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:17.471948    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d4fd4b59-2ecd-497b-a485-1e94101a9033-metrics-certs\") pod \"network-metrics-daemon-4sl67\" (UID: \"d4fd4b59-2ecd-497b-a485-1e94101a9033\") " pod="openshift-multus/network-metrics-daemon-4sl67"
Jul 13 18:50:17.472277 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:50:17.472138    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 13 18:50:17.472277 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:50:17.472222    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d4fd4b59-2ecd-497b-a485-1e94101a9033-metrics-certs podName:d4fd4b59-2ecd-497b-a485-1e94101a9033 nodeName:}" failed. No retries permitted until 2026-07-13 18:50:19.472203436 +0000 UTC m=+5.997336836 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d4fd4b59-2ecd-497b-a485-1e94101a9033-metrics-certs") pod "network-metrics-daemon-4sl67" (UID: "d4fd4b59-2ecd-497b-a485-1e94101a9033") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 13 18:50:17.573179 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:17.573136    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6bbc0748-e539-4679-a100-1ba45794129b-original-pull-secret\") pod \"global-pull-secret-syncer-qlzr5\" (UID: \"6bbc0748-e539-4679-a100-1ba45794129b\") " pod="kube-system/global-pull-secret-syncer-qlzr5"
Jul 13 18:50:17.573418 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:50:17.573310    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 13 18:50:17.573418 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:50:17.573385    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6bbc0748-e539-4679-a100-1ba45794129b-original-pull-secret podName:6bbc0748-e539-4679-a100-1ba45794129b nodeName:}" failed. No retries permitted until 2026-07-13 18:50:19.573362339 +0000 UTC m=+6.098495743 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6bbc0748-e539-4679-a100-1ba45794129b-original-pull-secret") pod "global-pull-secret-syncer-qlzr5" (UID: "6bbc0748-e539-4679-a100-1ba45794129b") : object "kube-system"/"original-pull-secret" not registered
Jul 13 18:50:17.674250 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:17.674202    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvvhf\" (UniqueName: \"kubernetes.io/projected/76869c1f-1537-4e3e-b7e2-654a08cd2376-kube-api-access-qvvhf\") pod \"network-check-target-5tgm8\" (UID: \"76869c1f-1537-4e3e-b7e2-654a08cd2376\") " pod="openshift-network-diagnostics/network-check-target-5tgm8"
Jul 13 18:50:17.674465 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:50:17.674444    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 13 18:50:17.674537 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:50:17.674470    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 13 18:50:17.674537 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:50:17.674485    2565 projected.go:196] Error preparing data for projected volume kube-api-access-qvvhf for pod openshift-network-diagnostics/network-check-target-5tgm8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 13 18:50:17.674654 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:50:17.674554    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/76869c1f-1537-4e3e-b7e2-654a08cd2376-kube-api-access-qvvhf podName:76869c1f-1537-4e3e-b7e2-654a08cd2376 nodeName:}" failed. No retries permitted until 2026-07-13 18:50:19.674534164 +0000 UTC m=+6.199667578 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-qvvhf" (UniqueName: "kubernetes.io/projected/76869c1f-1537-4e3e-b7e2-654a08cd2376-kube-api-access-qvvhf") pod "network-check-target-5tgm8" (UID: "76869c1f-1537-4e3e-b7e2-654a08cd2376") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 13 18:50:17.919206 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:17.919091    2565 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-12 18:45:16 +0000 UTC" deadline="2027-12-08 19:29:20.11451181 +0000 UTC"
Jul 13 18:50:17.919206 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:17.919145    2565 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12312h39m2.195372293s"
Jul 13 18:50:18.043039 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:18.042072    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5tgm8"
Jul 13 18:50:18.043039 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:50:18.042176    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5tgm8" podUID="76869c1f-1537-4e3e-b7e2-654a08cd2376"
Jul 13 18:50:18.043039 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:18.042038    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4sl67"
Jul 13 18:50:18.043039 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:50:18.042974    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4sl67" podUID="d4fd4b59-2ecd-497b-a485-1e94101a9033"
Jul 13 18:50:19.041762 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:19.041725    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qlzr5"
Jul 13 18:50:19.042272 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:50:19.041888    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qlzr5" podUID="6bbc0748-e539-4679-a100-1ba45794129b"
Jul 13 18:50:19.493589 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:19.493490    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d4fd4b59-2ecd-497b-a485-1e94101a9033-metrics-certs\") pod \"network-metrics-daemon-4sl67\" (UID: \"d4fd4b59-2ecd-497b-a485-1e94101a9033\") " pod="openshift-multus/network-metrics-daemon-4sl67"
Jul 13 18:50:19.493819 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:50:19.493689    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 13 18:50:19.493819 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:50:19.493786    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d4fd4b59-2ecd-497b-a485-1e94101a9033-metrics-certs podName:d4fd4b59-2ecd-497b-a485-1e94101a9033 nodeName:}" failed. No retries permitted until 2026-07-13 18:50:23.493743247 +0000 UTC m=+10.018876661 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d4fd4b59-2ecd-497b-a485-1e94101a9033-metrics-certs") pod "network-metrics-daemon-4sl67" (UID: "d4fd4b59-2ecd-497b-a485-1e94101a9033") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 13 18:50:19.594667 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:19.594624    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6bbc0748-e539-4679-a100-1ba45794129b-original-pull-secret\") pod \"global-pull-secret-syncer-qlzr5\" (UID: \"6bbc0748-e539-4679-a100-1ba45794129b\") " pod="kube-system/global-pull-secret-syncer-qlzr5"
Jul 13 18:50:19.594896 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:50:19.594808    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 13 18:50:19.594896 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:50:19.594877    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6bbc0748-e539-4679-a100-1ba45794129b-original-pull-secret podName:6bbc0748-e539-4679-a100-1ba45794129b nodeName:}" failed. No retries permitted until 2026-07-13 18:50:23.594858884 +0000 UTC m=+10.119992284 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6bbc0748-e539-4679-a100-1ba45794129b-original-pull-secret") pod "global-pull-secret-syncer-qlzr5" (UID: "6bbc0748-e539-4679-a100-1ba45794129b") : object "kube-system"/"original-pull-secret" not registered
Jul 13 18:50:19.695026 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:19.694982    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvvhf\" (UniqueName: \"kubernetes.io/projected/76869c1f-1537-4e3e-b7e2-654a08cd2376-kube-api-access-qvvhf\") pod \"network-check-target-5tgm8\" (UID: \"76869c1f-1537-4e3e-b7e2-654a08cd2376\") " pod="openshift-network-diagnostics/network-check-target-5tgm8"
Jul 13 18:50:19.695275 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:50:19.695244    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 13 18:50:19.695275 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:50:19.695266    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 13 18:50:19.695409 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:50:19.695280    2565 projected.go:196] Error preparing data for projected volume kube-api-access-qvvhf for pod openshift-network-diagnostics/network-check-target-5tgm8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 13 18:50:19.695409 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:50:19.695345    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/76869c1f-1537-4e3e-b7e2-654a08cd2376-kube-api-access-qvvhf podName:76869c1f-1537-4e3e-b7e2-654a08cd2376 nodeName:}" failed. No retries permitted until 2026-07-13 18:50:23.695325137 +0000 UTC m=+10.220458536 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-qvvhf" (UniqueName: "kubernetes.io/projected/76869c1f-1537-4e3e-b7e2-654a08cd2376-kube-api-access-qvvhf") pod "network-check-target-5tgm8" (UID: "76869c1f-1537-4e3e-b7e2-654a08cd2376") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 13 18:50:20.042955 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:20.042917    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5tgm8"
Jul 13 18:50:20.043463 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:50:20.043067    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5tgm8" podUID="76869c1f-1537-4e3e-b7e2-654a08cd2376"
Jul 13 18:50:20.043663 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:20.043602    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4sl67"
Jul 13 18:50:20.043809 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:50:20.043751    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4sl67" podUID="d4fd4b59-2ecd-497b-a485-1e94101a9033"
Jul 13 18:50:21.041878 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:21.041836    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qlzr5"
Jul 13 18:50:21.042085 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:50:21.042003    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qlzr5" podUID="6bbc0748-e539-4679-a100-1ba45794129b"
Jul 13 18:50:22.044665 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:22.044628    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5tgm8"
Jul 13 18:50:22.045185 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:50:22.044798    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5tgm8" podUID="76869c1f-1537-4e3e-b7e2-654a08cd2376"
Jul 13 18:50:22.045283 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:22.045262    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4sl67"
Jul 13 18:50:22.045473 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:50:22.045447    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4sl67" podUID="d4fd4b59-2ecd-497b-a485-1e94101a9033"
Jul 13 18:50:23.042720 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:23.042648    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qlzr5"
Jul 13 18:50:23.042931 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:50:23.042820    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qlzr5" podUID="6bbc0748-e539-4679-a100-1ba45794129b"
Jul 13 18:50:23.532083 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:23.531982    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d4fd4b59-2ecd-497b-a485-1e94101a9033-metrics-certs\") pod \"network-metrics-daemon-4sl67\" (UID: \"d4fd4b59-2ecd-497b-a485-1e94101a9033\") " pod="openshift-multus/network-metrics-daemon-4sl67"
Jul 13 18:50:23.532563 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:50:23.532205    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 13 18:50:23.532563 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:50:23.532260    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d4fd4b59-2ecd-497b-a485-1e94101a9033-metrics-certs podName:d4fd4b59-2ecd-497b-a485-1e94101a9033 nodeName:}" failed. No retries permitted until 2026-07-13 18:50:31.532245737 +0000 UTC m=+18.057379133 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d4fd4b59-2ecd-497b-a485-1e94101a9033-metrics-certs") pod "network-metrics-daemon-4sl67" (UID: "d4fd4b59-2ecd-497b-a485-1e94101a9033") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 13 18:50:23.632896 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:23.632852    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6bbc0748-e539-4679-a100-1ba45794129b-original-pull-secret\") pod \"global-pull-secret-syncer-qlzr5\" (UID: \"6bbc0748-e539-4679-a100-1ba45794129b\") " pod="kube-system/global-pull-secret-syncer-qlzr5"
Jul 13 18:50:23.633112 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:50:23.633064    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 13 18:50:23.633194 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:50:23.633149    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6bbc0748-e539-4679-a100-1ba45794129b-original-pull-secret podName:6bbc0748-e539-4679-a100-1ba45794129b nodeName:}" failed. No retries permitted until 2026-07-13 18:50:31.633127973 +0000 UTC m=+18.158261376 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6bbc0748-e539-4679-a100-1ba45794129b-original-pull-secret") pod "global-pull-secret-syncer-qlzr5" (UID: "6bbc0748-e539-4679-a100-1ba45794129b") : object "kube-system"/"original-pull-secret" not registered
Jul 13 18:50:23.734029 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:23.733983    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvvhf\" (UniqueName: \"kubernetes.io/projected/76869c1f-1537-4e3e-b7e2-654a08cd2376-kube-api-access-qvvhf\") pod \"network-check-target-5tgm8\" (UID: \"76869c1f-1537-4e3e-b7e2-654a08cd2376\") " pod="openshift-network-diagnostics/network-check-target-5tgm8"
Jul 13 18:50:23.734216 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:50:23.734192    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 13 18:50:23.734266 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:50:23.734221    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 13 18:50:23.734266 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:50:23.734236    2565 projected.go:196] Error preparing data for projected volume kube-api-access-qvvhf for pod openshift-network-diagnostics/network-check-target-5tgm8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 13 18:50:23.734356 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:50:23.734293    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/76869c1f-1537-4e3e-b7e2-654a08cd2376-kube-api-access-qvvhf podName:76869c1f-1537-4e3e-b7e2-654a08cd2376 nodeName:}" failed. No retries permitted until 2026-07-13 18:50:31.73427751 +0000 UTC m=+18.259410908 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-qvvhf" (UniqueName: "kubernetes.io/projected/76869c1f-1537-4e3e-b7e2-654a08cd2376-kube-api-access-qvvhf") pod "network-check-target-5tgm8" (UID: "76869c1f-1537-4e3e-b7e2-654a08cd2376") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 13 18:50:24.044149 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:24.044112    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5tgm8"
Jul 13 18:50:24.044341 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:50:24.044244    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5tgm8" podUID="76869c1f-1537-4e3e-b7e2-654a08cd2376"
Jul 13 18:50:24.044723 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:24.044681    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4sl67"
Jul 13 18:50:24.044836 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:50:24.044816    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4sl67" podUID="d4fd4b59-2ecd-497b-a485-1e94101a9033"
Jul 13 18:50:25.042530 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:25.042491    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qlzr5"
Jul 13 18:50:25.042982 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:50:25.042635    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qlzr5" podUID="6bbc0748-e539-4679-a100-1ba45794129b"
Jul 13 18:50:26.043002 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:26.042435    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5tgm8"
Jul 13 18:50:26.043002 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:50:26.042583    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5tgm8" podUID="76869c1f-1537-4e3e-b7e2-654a08cd2376"
Jul 13 18:50:26.043824 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:26.043629    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4sl67"
Jul 13 18:50:26.043824 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:50:26.043780    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4sl67" podUID="d4fd4b59-2ecd-497b-a485-1e94101a9033"
Jul 13 18:50:27.041734 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:27.041682    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qlzr5"
Jul 13 18:50:27.041902 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:50:27.041827    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qlzr5" podUID="6bbc0748-e539-4679-a100-1ba45794129b"
Jul 13 18:50:28.042606 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:28.042570    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5tgm8"
Jul 13 18:50:28.043078 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:28.042568    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4sl67"
Jul 13 18:50:28.043078 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:50:28.042728    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5tgm8" podUID="76869c1f-1537-4e3e-b7e2-654a08cd2376"
Jul 13 18:50:28.043078 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:50:28.042819    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4sl67" podUID="d4fd4b59-2ecd-497b-a485-1e94101a9033"
Jul 13 18:50:29.042488 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:29.042445    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qlzr5"
Jul 13 18:50:29.042635 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:50:29.042562    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qlzr5" podUID="6bbc0748-e539-4679-a100-1ba45794129b"
Jul 13 18:50:30.042637 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:30.042597    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4sl67"
Jul 13 18:50:30.043093 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:30.042607    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5tgm8"
Jul 13 18:50:30.043093 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:50:30.042778    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4sl67" podUID="d4fd4b59-2ecd-497b-a485-1e94101a9033"
Jul 13 18:50:30.043093 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:50:30.042843    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5tgm8" podUID="76869c1f-1537-4e3e-b7e2-654a08cd2376"
Jul 13 18:50:31.042758 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:31.042723    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qlzr5"
Jul 13 18:50:31.043325 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:50:31.042843    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qlzr5" podUID="6bbc0748-e539-4679-a100-1ba45794129b"
Jul 13 18:50:31.599921 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:31.598996    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d4fd4b59-2ecd-497b-a485-1e94101a9033-metrics-certs\") pod \"network-metrics-daemon-4sl67\" (UID: \"d4fd4b59-2ecd-497b-a485-1e94101a9033\") " pod="openshift-multus/network-metrics-daemon-4sl67"
Jul 13 18:50:31.599921 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:50:31.599135    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 13 18:50:31.599921 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:50:31.599222    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d4fd4b59-2ecd-497b-a485-1e94101a9033-metrics-certs podName:d4fd4b59-2ecd-497b-a485-1e94101a9033 nodeName:}" failed. No retries permitted until 2026-07-13 18:50:47.599198368 +0000 UTC m=+34.124331766 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d4fd4b59-2ecd-497b-a485-1e94101a9033-metrics-certs") pod "network-metrics-daemon-4sl67" (UID: "d4fd4b59-2ecd-497b-a485-1e94101a9033") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 13 18:50:31.699463 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:31.699421    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6bbc0748-e539-4679-a100-1ba45794129b-original-pull-secret\") pod \"global-pull-secret-syncer-qlzr5\" (UID: \"6bbc0748-e539-4679-a100-1ba45794129b\") " pod="kube-system/global-pull-secret-syncer-qlzr5"
Jul 13 18:50:31.699675 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:50:31.699609    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 13 18:50:31.699785 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:50:31.699693    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6bbc0748-e539-4679-a100-1ba45794129b-original-pull-secret podName:6bbc0748-e539-4679-a100-1ba45794129b nodeName:}" failed. No retries permitted until 2026-07-13 18:50:47.699671621 +0000 UTC m=+34.224805016 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6bbc0748-e539-4679-a100-1ba45794129b-original-pull-secret") pod "global-pull-secret-syncer-qlzr5" (UID: "6bbc0748-e539-4679-a100-1ba45794129b") : object "kube-system"/"original-pull-secret" not registered
Jul 13 18:50:31.799851 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:31.799808    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvvhf\" (UniqueName: \"kubernetes.io/projected/76869c1f-1537-4e3e-b7e2-654a08cd2376-kube-api-access-qvvhf\") pod \"network-check-target-5tgm8\" (UID: \"76869c1f-1537-4e3e-b7e2-654a08cd2376\") " pod="openshift-network-diagnostics/network-check-target-5tgm8"
Jul 13 18:50:31.800052 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:50:31.799961    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 13 18:50:31.800052 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:50:31.799989    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 13 18:50:31.800052 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:50:31.800003    2565 projected.go:196] Error preparing data for projected volume kube-api-access-qvvhf for pod openshift-network-diagnostics/network-check-target-5tgm8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 13 18:50:31.800211 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:50:31.800071    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/76869c1f-1537-4e3e-b7e2-654a08cd2376-kube-api-access-qvvhf podName:76869c1f-1537-4e3e-b7e2-654a08cd2376 nodeName:}" failed. No retries permitted until 2026-07-13 18:50:47.800051424 +0000 UTC m=+34.325184828 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-qvvhf" (UniqueName: "kubernetes.io/projected/76869c1f-1537-4e3e-b7e2-654a08cd2376-kube-api-access-qvvhf") pod "network-check-target-5tgm8" (UID: "76869c1f-1537-4e3e-b7e2-654a08cd2376") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 13 18:50:32.042310 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:32.042210    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5tgm8"
Jul 13 18:50:32.042490 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:32.042215    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4sl67"
Jul 13 18:50:32.042490 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:50:32.042357    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5tgm8" podUID="76869c1f-1537-4e3e-b7e2-654a08cd2376"
Jul 13 18:50:32.042490 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:50:32.042423    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4sl67" podUID="d4fd4b59-2ecd-497b-a485-1e94101a9033"
Jul 13 18:50:33.042348 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:33.042308    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qlzr5"
Jul 13 18:50:33.042799 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:50:33.042434    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qlzr5" podUID="6bbc0748-e539-4679-a100-1ba45794129b"
Jul 13 18:50:34.045419 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:34.045336    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4sl67"
Jul 13 18:50:34.045419 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:34.045370    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5tgm8"
Jul 13 18:50:34.045927 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:50:34.045467    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4sl67" podUID="d4fd4b59-2ecd-497b-a485-1e94101a9033"
Jul 13 18:50:34.045927 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:50:34.045601    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5tgm8" podUID="76869c1f-1537-4e3e-b7e2-654a08cd2376"
Jul 13 18:50:35.042756 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:35.042719    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qlzr5"
Jul 13 18:50:35.043064 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:50:35.042831    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qlzr5" podUID="6bbc0748-e539-4679-a100-1ba45794129b"
Jul 13 18:50:36.046196 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:36.046165    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4sl67"
Jul 13 18:50:36.046679 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:36.046178    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5tgm8"
Jul 13 18:50:36.046679 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:50:36.046292    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4sl67" podUID="d4fd4b59-2ecd-497b-a485-1e94101a9033"
Jul 13 18:50:36.046679 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:50:36.046382    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5tgm8" podUID="76869c1f-1537-4e3e-b7e2-654a08cd2376"
Jul 13 18:50:37.042609 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:37.042574    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qlzr5"
Jul 13 18:50:37.042832 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:50:37.042683    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qlzr5" podUID="6bbc0748-e539-4679-a100-1ba45794129b"
Jul 13 18:50:38.045848 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:38.042561    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4sl67"
Jul 13 18:50:38.045848 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:50:38.042964    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4sl67" podUID="d4fd4b59-2ecd-497b-a485-1e94101a9033"
Jul 13 18:50:38.045848 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:38.043427    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5tgm8"
Jul 13 18:50:38.045848 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:50:38.043525    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5tgm8" podUID="76869c1f-1537-4e3e-b7e2-654a08cd2376"
Jul 13 18:50:38.131320 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:38.131281    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-gwvg9" event={"ID":"3a32830b-434d-4118-a91c-9dfdd29dcfa6","Type":"ContainerStarted","Data":"2d2b2048087f725f0d41b077b56f11ee6939008d24b4ef891449579f4dd099b1"}
Jul 13 18:50:38.143243 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:38.143166    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-n4lpd" event={"ID":"9dccc391-865f-45a2-9004-85e3830f39be","Type":"ContainerStarted","Data":"1ed19e525bd522ae97cf86b9c07176c561033f59edc1b4d08429e359c82644e2"}
Jul 13 18:50:38.145130 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:38.145074    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2bscc_2f0655bd-4a09-4314-8c1b-e30191eaec0e/ovn-acl-logging/0.log"
Jul 13 18:50:38.145559 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:38.145525    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2bscc" event={"ID":"2f0655bd-4a09-4314-8c1b-e30191eaec0e","Type":"ContainerStarted","Data":"6e838bcdb9dcb67f14b0ff97ac1bfdcb1c8022ca383e1cfd9475b10d8fa4a88b"}
Jul 13 18:50:38.145642 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:38.145607    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2bscc" event={"ID":"2f0655bd-4a09-4314-8c1b-e30191eaec0e","Type":"ContainerStarted","Data":"d7d9c87e3c17455ad756ea75374885535481c686fdd742c3d15e97022a8a96ea"}
Jul 13 18:50:38.151692 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:38.151652    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kgrbw" event={"ID":"6ca2c2d9-346a-48c4-af31-ab3b2e66539f","Type":"ContainerStarted","Data":"a4462cd5f7c9d386c4b902b6134f67e062730bbf0eafbe6b3501c7f5643d9b7b"}
Jul 13 18:50:38.157143 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:38.157108    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-qtpwz" event={"ID":"ef958600-65d1-473a-b2dd-397c1dbbc8da","Type":"ContainerStarted","Data":"943e61923411098ef30d1eee5fd1149707fcb7cf9457bed5510d6ec66a5ac84b"}
Jul 13 18:50:38.169936 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:38.169805    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-gwvg9" podStartSLOduration=2.248701811 podStartE2EDuration="23.169786181s" totalImagesPullingTime="20.92108437s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 18:50:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:50:38.169615534 +0000 UTC m=+24.694748950" watchObservedRunningTime="2026-07-13 18:50:38.169786181 +0000 UTC m=+24.694919597"
Jul 13 18:50:38.213176 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:38.212963    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-n4lpd" podStartSLOduration=3.273698325 podStartE2EDuration="24.212941649s" totalImagesPullingTime="20.939243324s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 18:50:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:50:38.212722169 +0000 UTC m=+24.737855604" watchObservedRunningTime="2026-07-13 18:50:38.212941649 +0000 UTC m=+24.738075066"
Jul 13 18:50:38.261166 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:38.261118    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-qtpwz" podStartSLOduration=3.256450933 podStartE2EDuration="24.261102218s" totalImagesPullingTime="21.004651285s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 18:50:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:50:38.260659276 +0000 UTC m=+24.785792691" watchObservedRunningTime="2026-07-13 18:50:38.261102218 +0000 UTC m=+24.786235635"
Jul 13 18:50:39.042668 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:39.042623    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qlzr5"
Jul 13 18:50:39.042905 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:50:39.042773    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qlzr5" podUID="6bbc0748-e539-4679-a100-1ba45794129b"
Jul 13 18:50:39.160517 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:39.160480    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-psj7d" event={"ID":"34d8b129-0c9e-4cda-8669-d7595f3d01af","Type":"ContainerStarted","Data":"4d7c27b5365ea2406552fd210f1d1a2c57bca4b3d9821cf3950b78e25f6aea63"}
Jul 13 18:50:39.162380 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:39.162348    2565 generic.go:356] "Generic (PLEG): container finished" podID="fce2836fb78b250794cad725b074cef4" containerID="539cf4d7d8a13ae6ab91a3ba5b4d4d812208abfe61d21fd22b0ee548bab6b10d" exitCode=0
Jul 13 18:50:39.162527 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:39.162450    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-215.ec2.internal" event={"ID":"fce2836fb78b250794cad725b074cef4","Type":"ContainerDied","Data":"539cf4d7d8a13ae6ab91a3ba5b4d4d812208abfe61d21fd22b0ee548bab6b10d"}
Jul 13 18:50:39.164125 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:39.164102    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-zcq5p" event={"ID":"6ae58b1e-3aaf-45dc-a741-2fb895468ca7","Type":"ContainerStarted","Data":"1b7c8f3cd83c8e893ee6f3f473f4732f087314e0a014beaa1b7b865d3e8d9cbd"}
Jul 13 18:50:39.165491 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:39.165468    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-k2vg4" event={"ID":"b7240a5a-c704-49a7-b152-bdeb36cee9d4","Type":"ContainerStarted","Data":"f4082e860d0696c82ae84eaa35dfd74921984ba1dc0f474eee5384bcc538cb6a"}
Jul 13 18:50:39.168410 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:39.168393    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2bscc_2f0655bd-4a09-4314-8c1b-e30191eaec0e/ovn-acl-logging/0.log"
Jul 13 18:50:39.168786 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:39.168768    2565 generic.go:356] "Generic (PLEG): container finished" podID="2f0655bd-4a09-4314-8c1b-e30191eaec0e" containerID="6e838bcdb9dcb67f14b0ff97ac1bfdcb1c8022ca383e1cfd9475b10d8fa4a88b" exitCode=1
Jul 13 18:50:39.168854 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:39.168810    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2bscc" event={"ID":"2f0655bd-4a09-4314-8c1b-e30191eaec0e","Type":"ContainerDied","Data":"6e838bcdb9dcb67f14b0ff97ac1bfdcb1c8022ca383e1cfd9475b10d8fa4a88b"}
Jul 13 18:50:39.168910 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:39.168880    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2bscc" event={"ID":"2f0655bd-4a09-4314-8c1b-e30191eaec0e","Type":"ContainerStarted","Data":"43b11319be236f1eec00386602f42cfd95b021ffc42dc1a084bc2d914cbe6302"}
Jul 13 18:50:39.168963 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:39.168918    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2bscc" event={"ID":"2f0655bd-4a09-4314-8c1b-e30191eaec0e","Type":"ContainerStarted","Data":"4285397e8387bffc05d78548e73c67f5b81d3745a581b3f6ec5d9d2a45baa450"}
Jul 13 18:50:39.169013 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:39.168964    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2bscc" event={"ID":"2f0655bd-4a09-4314-8c1b-e30191eaec0e","Type":"ContainerStarted","Data":"c1fea5705c5ad94c7229650177f251579cef53d694d1edabaa5aaab0b4c41f1b"}
Jul 13 18:50:39.169013 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:39.168995    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2bscc" event={"ID":"2f0655bd-4a09-4314-8c1b-e30191eaec0e","Type":"ContainerStarted","Data":"965235fba6b146e3666e49219e4819b1777dd2ba51035d46618c0cd36449b741"}
Jul 13 18:50:39.170327 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:39.170300    2565 generic.go:356] "Generic (PLEG): container finished" podID="6ca2c2d9-346a-48c4-af31-ab3b2e66539f" containerID="a4462cd5f7c9d386c4b902b6134f67e062730bbf0eafbe6b3501c7f5643d9b7b" exitCode=0
Jul 13 18:50:39.170442 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:39.170382    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kgrbw" event={"ID":"6ca2c2d9-346a-48c4-af31-ab3b2e66539f","Type":"ContainerDied","Data":"a4462cd5f7c9d386c4b902b6134f67e062730bbf0eafbe6b3501c7f5643d9b7b"}
Jul 13 18:50:39.195780 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:39.195731    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-k2vg4" podStartSLOduration=4.2564256799999995 podStartE2EDuration="25.195688477s" totalImagesPullingTime="20.939262797s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 18:50:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:50:39.195073798 +0000 UTC m=+25.720207216" watchObservedRunningTime="2026-07-13 18:50:39.195688477 +0000 UTC m=+25.720821893"
Jul 13 18:50:39.234366 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:39.234304    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-zcq5p" podStartSLOduration=4.293341389 podStartE2EDuration="25.23428568s" totalImagesPullingTime="20.940944291s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 18:50:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:50:39.233906109 +0000 UTC m=+25.759039546" watchObservedRunningTime="2026-07-13 18:50:39.23428568 +0000 UTC m=+25.759419095"
Jul 13 18:50:39.809511 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:39.809482    2565 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 13 18:50:39.907096 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:39.906958    2565 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-13T18:50:39.809504137Z","UUID":"d51cb5be-86fb-4c8f-bc8c-f19b1f02e0cb","Handler":null,"Name":"","Endpoint":""}
Jul 13 18:50:39.908722 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:39.908684    2565 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 13 18:50:39.908851 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:39.908774    2565 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 13 18:50:40.046818 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:40.046734    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4sl67"
Jul 13 18:50:40.046818 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:40.046753    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5tgm8"
Jul 13 18:50:40.047099 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:50:40.046855    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5tgm8" podUID="76869c1f-1537-4e3e-b7e2-654a08cd2376"
Jul 13 18:50:40.047099 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:50:40.046977    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4sl67" podUID="d4fd4b59-2ecd-497b-a485-1e94101a9033"
Jul 13 18:50:40.175301 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:40.175257    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-psj7d" event={"ID":"34d8b129-0c9e-4cda-8669-d7595f3d01af","Type":"ContainerStarted","Data":"d5260d4b0cfca2b19c55d61a0c135ade6c1e17982d60f9d1f77ddaeae0e7e0e5"}
Jul 13 18:50:40.177318 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:40.177283    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-215.ec2.internal" event={"ID":"fce2836fb78b250794cad725b074cef4","Type":"ContainerStarted","Data":"586f689a9a0aa29f07f44e245779acc5cd40f549cabc07d04550ee8f7d9594c3"}
Jul 13 18:50:40.178947 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:40.178914    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-pxrrr" event={"ID":"10b9cbe5-c0a3-4d27-af48-2cebde3572bc","Type":"ContainerStarted","Data":"43714ec0bb552c7491471deaafd91376344ed7f4cf2634021cf305af16ce9825"}
Jul 13 18:50:40.194651 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:40.194584    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-215.ec2.internal" podStartSLOduration=25.194561995 podStartE2EDuration="25.194561995s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 18:50:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:50:40.193899939 +0000 UTC m=+26.719033355" watchObservedRunningTime="2026-07-13 18:50:40.194561995 +0000 UTC m=+26.719695413"
Jul 13 18:50:40.210040 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:40.209983    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-pxrrr" podStartSLOduration=5.29005126 podStartE2EDuration="26.209962414s" totalImagesPullingTime="20.919911154s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 18:50:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:50:40.209600467 +0000 UTC m=+26.734733910" watchObservedRunningTime="2026-07-13 18:50:40.209962414 +0000 UTC m=+26.735095833"
Jul 13 18:50:41.042362 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:41.042081    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qlzr5"
Jul 13 18:50:41.042539 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:50:41.042419    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qlzr5" podUID="6bbc0748-e539-4679-a100-1ba45794129b"
Jul 13 18:50:41.185910 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:41.185886    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2bscc_2f0655bd-4a09-4314-8c1b-e30191eaec0e/ovn-acl-logging/0.log"
Jul 13 18:50:41.186300 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:41.186271    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2bscc" event={"ID":"2f0655bd-4a09-4314-8c1b-e30191eaec0e","Type":"ContainerStarted","Data":"2b73da4bfe49bb4cf13188274b55dd5e2cd1387dd9bf9fa0f4f0f9a532878507"}
Jul 13 18:50:42.042268 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:42.042180    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5tgm8"
Jul 13 18:50:42.042448 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:50:42.042314    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5tgm8" podUID="76869c1f-1537-4e3e-b7e2-654a08cd2376"
Jul 13 18:50:42.042775 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:42.042184    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4sl67"
Jul 13 18:50:42.042877 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:50:42.042843    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4sl67" podUID="d4fd4b59-2ecd-497b-a485-1e94101a9033"
Jul 13 18:50:42.192570 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:42.192028    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-psj7d" event={"ID":"34d8b129-0c9e-4cda-8669-d7595f3d01af","Type":"ContainerStarted","Data":"97dc410cd7122a0b39145568d813c571f055baf9c9d81f697d00a87bc872cf1f"}
Jul 13 18:50:42.214967 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:42.214915    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-psj7d" podStartSLOduration=3.527480326 podStartE2EDuration="27.214900476s" totalImagesPullingTime="23.68742015s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 18:50:15 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:50:42.214746212 +0000 UTC m=+28.739879661" watchObservedRunningTime="2026-07-13 18:50:42.214900476 +0000 UTC m=+28.740033891"
Jul 13 18:50:42.254418 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:42.254379    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-n4lpd"
Jul 13 18:50:42.255496 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:42.255470    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-n4lpd"
Jul 13 18:50:43.042213 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:43.042172    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qlzr5"
Jul 13 18:50:43.042423 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:50:43.042301    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qlzr5" podUID="6bbc0748-e539-4679-a100-1ba45794129b"
Jul 13 18:50:43.195104 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:43.195016    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-n4lpd"
Jul 13 18:50:43.195542 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:43.195466    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-n4lpd"
Jul 13 18:50:44.045624 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:44.045437    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4sl67"
Jul 13 18:50:44.045785 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:44.045476    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5tgm8"
Jul 13 18:50:44.045785 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:50:44.045747    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4sl67" podUID="d4fd4b59-2ecd-497b-a485-1e94101a9033"
Jul 13 18:50:44.045850 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:50:44.045779    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5tgm8" podUID="76869c1f-1537-4e3e-b7e2-654a08cd2376"
Jul 13 18:50:44.200390 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:44.200306    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2bscc_2f0655bd-4a09-4314-8c1b-e30191eaec0e/ovn-acl-logging/0.log"
Jul 13 18:50:44.201055 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:44.200680    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2bscc" event={"ID":"2f0655bd-4a09-4314-8c1b-e30191eaec0e","Type":"ContainerStarted","Data":"2994d22162bfb4791419e6a5606f87f9ae5c0cf69cbccc999ee91a7eebc0a412"}
Jul 13 18:50:44.201055 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:44.200995    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-2bscc"
Jul 13 18:50:44.201159 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:44.201085    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-2bscc"
Jul 13 18:50:44.201228 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:44.201211    2565 scope.go:117] "RemoveContainer" containerID="6e838bcdb9dcb67f14b0ff97ac1bfdcb1c8022ca383e1cfd9475b10d8fa4a88b"
Jul 13 18:50:44.202811 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:44.202788    2565 generic.go:356] "Generic (PLEG): container finished" podID="6ca2c2d9-346a-48c4-af31-ab3b2e66539f" containerID="a42b0ad3257a2cd1a9cc2cbd2318d0751776921d5eaf816038a57930faedc03b" exitCode=0
Jul 13 18:50:44.202934 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:44.202873    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kgrbw" event={"ID":"6ca2c2d9-346a-48c4-af31-ab3b2e66539f","Type":"ContainerDied","Data":"a42b0ad3257a2cd1a9cc2cbd2318d0751776921d5eaf816038a57930faedc03b"}
Jul 13 18:50:44.218577 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:44.218548    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2bscc"
Jul 13 18:50:45.041987 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:45.041950    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qlzr5"
Jul 13 18:50:45.042191 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:50:45.042069    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qlzr5" podUID="6bbc0748-e539-4679-a100-1ba45794129b"
Jul 13 18:50:45.209465 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:45.209440    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2bscc_2f0655bd-4a09-4314-8c1b-e30191eaec0e/ovn-acl-logging/0.log"
Jul 13 18:50:45.210109 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:45.210078    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2bscc" event={"ID":"2f0655bd-4a09-4314-8c1b-e30191eaec0e","Type":"ContainerStarted","Data":"6ea04a0100c1a69fa2c251b7323e282a038826be918d5d633e708857b59aa390"}
Jul 13 18:50:45.210189 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:45.210169    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-2bscc"
Jul 13 18:50:45.226588 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:45.226562    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2bscc"
Jul 13 18:50:45.243872 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:45.243814    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-2bscc" podStartSLOduration=10.25453132 podStartE2EDuration="31.24379102s" totalImagesPullingTime="20.9892597s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 18:50:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:50:45.2436768 +0000 UTC m=+31.768810216" watchObservedRunningTime="2026-07-13 18:50:45.24379102 +0000 UTC m=+31.768924466"
Jul 13 18:50:45.512100 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:45.511924    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-qlzr5"]
Jul 13 18:50:45.512251 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:45.512207    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qlzr5"
Jul 13 18:50:45.512304 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:50:45.512287    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qlzr5" podUID="6bbc0748-e539-4679-a100-1ba45794129b"
Jul 13 18:50:45.514231 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:45.514196    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-5tgm8"]
Jul 13 18:50:45.514365 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:45.514318    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5tgm8"
Jul 13 18:50:45.514426 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:50:45.514408    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5tgm8" podUID="76869c1f-1537-4e3e-b7e2-654a08cd2376"
Jul 13 18:50:45.528065 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:45.528031    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-4sl67"]
Jul 13 18:50:45.528217 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:45.528185    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4sl67"
Jul 13 18:50:45.528338 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:50:45.528315    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4sl67" podUID="d4fd4b59-2ecd-497b-a485-1e94101a9033"
Jul 13 18:50:46.214722 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:46.214669    2565 generic.go:356] "Generic (PLEG): container finished" podID="6ca2c2d9-346a-48c4-af31-ab3b2e66539f" containerID="1fa41993a90853ee016b195e16f58c08dea89dfd0a58ccdc2b86b3a7743a87b7" exitCode=0
Jul 13 18:50:46.215358 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:46.214733    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kgrbw" event={"ID":"6ca2c2d9-346a-48c4-af31-ab3b2e66539f","Type":"ContainerDied","Data":"1fa41993a90853ee016b195e16f58c08dea89dfd0a58ccdc2b86b3a7743a87b7"}
Jul 13 18:50:47.042539 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:47.042503    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5tgm8"
Jul 13 18:50:47.042539 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:47.042532    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qlzr5"
Jul 13 18:50:47.042794 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:47.042508    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4sl67"
Jul 13 18:50:47.042794 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:50:47.042609    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5tgm8" podUID="76869c1f-1537-4e3e-b7e2-654a08cd2376"
Jul 13 18:50:47.042794 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:50:47.042738    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4sl67" podUID="d4fd4b59-2ecd-497b-a485-1e94101a9033"
Jul 13 18:50:47.042919 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:50:47.042824    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qlzr5" podUID="6bbc0748-e539-4679-a100-1ba45794129b"
Jul 13 18:50:47.219786 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:47.219749    2565 generic.go:356] "Generic (PLEG): container finished" podID="6ca2c2d9-346a-48c4-af31-ab3b2e66539f" containerID="f379fbdc6a067e272ad461ad146f96470c9a7057d6d37781c853c6c6c11dd10b" exitCode=0
Jul 13 18:50:47.220280 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:47.219836    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kgrbw" event={"ID":"6ca2c2d9-346a-48c4-af31-ab3b2e66539f","Type":"ContainerDied","Data":"f379fbdc6a067e272ad461ad146f96470c9a7057d6d37781c853c6c6c11dd10b"}
Jul 13 18:50:47.622894 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:47.622853    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d4fd4b59-2ecd-497b-a485-1e94101a9033-metrics-certs\") pod \"network-metrics-daemon-4sl67\" (UID: \"d4fd4b59-2ecd-497b-a485-1e94101a9033\") " pod="openshift-multus/network-metrics-daemon-4sl67"
Jul 13 18:50:47.623064 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:50:47.623006    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 13 18:50:47.623116 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:50:47.623073    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d4fd4b59-2ecd-497b-a485-1e94101a9033-metrics-certs podName:d4fd4b59-2ecd-497b-a485-1e94101a9033 nodeName:}" failed. No retries permitted until 2026-07-13 18:51:19.623056468 +0000 UTC m=+66.148189866 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d4fd4b59-2ecd-497b-a485-1e94101a9033-metrics-certs") pod "network-metrics-daemon-4sl67" (UID: "d4fd4b59-2ecd-497b-a485-1e94101a9033") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 13 18:50:47.724242 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:47.724200    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6bbc0748-e539-4679-a100-1ba45794129b-original-pull-secret\") pod \"global-pull-secret-syncer-qlzr5\" (UID: \"6bbc0748-e539-4679-a100-1ba45794129b\") " pod="kube-system/global-pull-secret-syncer-qlzr5"
Jul 13 18:50:47.724424 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:50:47.724341    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 13 18:50:47.724424 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:50:47.724401    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6bbc0748-e539-4679-a100-1ba45794129b-original-pull-secret podName:6bbc0748-e539-4679-a100-1ba45794129b nodeName:}" failed. No retries permitted until 2026-07-13 18:51:19.724386327 +0000 UTC m=+66.249519720 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6bbc0748-e539-4679-a100-1ba45794129b-original-pull-secret") pod "global-pull-secret-syncer-qlzr5" (UID: "6bbc0748-e539-4679-a100-1ba45794129b") : object "kube-system"/"original-pull-secret" not registered
Jul 13 18:50:47.825058 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:47.825019    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvvhf\" (UniqueName: \"kubernetes.io/projected/76869c1f-1537-4e3e-b7e2-654a08cd2376-kube-api-access-qvvhf\") pod \"network-check-target-5tgm8\" (UID: \"76869c1f-1537-4e3e-b7e2-654a08cd2376\") " pod="openshift-network-diagnostics/network-check-target-5tgm8"
Jul 13 18:50:47.825242 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:50:47.825220    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 13 18:50:47.825302 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:50:47.825250    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 13 18:50:47.825302 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:50:47.825261    2565 projected.go:196] Error preparing data for projected volume kube-api-access-qvvhf for pod openshift-network-diagnostics/network-check-target-5tgm8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 13 18:50:47.825405 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:50:47.825318    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/76869c1f-1537-4e3e-b7e2-654a08cd2376-kube-api-access-qvvhf podName:76869c1f-1537-4e3e-b7e2-654a08cd2376 nodeName:}" failed. No retries permitted until 2026-07-13 18:51:19.825302249 +0000 UTC m=+66.350435654 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-qvvhf" (UniqueName: "kubernetes.io/projected/76869c1f-1537-4e3e-b7e2-654a08cd2376-kube-api-access-qvvhf") pod "network-check-target-5tgm8" (UID: "76869c1f-1537-4e3e-b7e2-654a08cd2376") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 13 18:50:49.041836 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:49.041794    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4sl67"
Jul 13 18:50:49.041836 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:49.041818    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5tgm8"
Jul 13 18:50:49.041836 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:49.041840    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qlzr5"
Jul 13 18:50:49.042682 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:50:49.041939    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4sl67" podUID="d4fd4b59-2ecd-497b-a485-1e94101a9033"
Jul 13 18:50:49.042682 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:50:49.042053    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5tgm8" podUID="76869c1f-1537-4e3e-b7e2-654a08cd2376"
Jul 13 18:50:49.042682 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:50:49.042137    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qlzr5" podUID="6bbc0748-e539-4679-a100-1ba45794129b"
Jul 13 18:50:51.041895 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:51.041857    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qlzr5"
Jul 13 18:50:51.042378 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:51.041897    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5tgm8"
Jul 13 18:50:51.042378 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:51.041902    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4sl67"
Jul 13 18:50:51.042378 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:50:51.041996    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qlzr5" podUID="6bbc0748-e539-4679-a100-1ba45794129b"
Jul 13 18:50:51.042378 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:50:51.042093    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5tgm8" podUID="76869c1f-1537-4e3e-b7e2-654a08cd2376"
Jul 13 18:50:51.042378 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:50:51.042174    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4sl67" podUID="d4fd4b59-2ecd-497b-a485-1e94101a9033"
Jul 13 18:50:51.794580 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:51.794547    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-215.ec2.internal" event="NodeReady"
Jul 13 18:50:51.794757 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:51.794688    2565 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 13 18:50:51.859576 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:51.859534    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-jw7lc"]
Jul 13 18:50:51.861429 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:51.861400    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-4rt5g"]
Jul 13 18:50:51.861565 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:51.861531    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-jw7lc"
Jul 13 18:50:51.863079 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:51.863055    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-4rt5g"
Jul 13 18:50:51.866809 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:51.866784    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 13 18:50:51.866962 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:51.866831    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 13 18:50:51.866962 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:51.866784    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-88cff\""
Jul 13 18:50:51.867285 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:51.867247    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-bq5zm\""
Jul 13 18:50:51.867427 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:51.867352    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 13 18:50:51.867517 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:51.867502    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 13 18:50:51.867808 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:51.867785    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 13 18:50:51.878974 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:51.878945    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-4rt5g"]
Jul 13 18:50:51.893336 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:51.893307    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-jw7lc"]
Jul 13 18:50:51.956521 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:51.956397    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2ddf2ee1-c9d7-4f30-aa52-b94c6d250c70-cert\") pod \"ingress-canary-4rt5g\" (UID: \"2ddf2ee1-c9d7-4f30-aa52-b94c6d250c70\") " pod="openshift-ingress-canary/ingress-canary-4rt5g"
Jul 13 18:50:51.956521 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:51.956465    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqg6t\" (UniqueName: \"kubernetes.io/projected/5a144761-c470-4ac9-95cb-a629c9aac64a-kube-api-access-lqg6t\") pod \"dns-default-jw7lc\" (UID: \"5a144761-c470-4ac9-95cb-a629c9aac64a\") " pod="openshift-dns/dns-default-jw7lc"
Jul 13 18:50:51.956743 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:51.956524    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4kj9k\" (UniqueName: \"kubernetes.io/projected/2ddf2ee1-c9d7-4f30-aa52-b94c6d250c70-kube-api-access-4kj9k\") pod \"ingress-canary-4rt5g\" (UID: \"2ddf2ee1-c9d7-4f30-aa52-b94c6d250c70\") " pod="openshift-ingress-canary/ingress-canary-4rt5g"
Jul 13 18:50:51.956743 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:51.956557    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5a144761-c470-4ac9-95cb-a629c9aac64a-config-volume\") pod \"dns-default-jw7lc\" (UID: \"5a144761-c470-4ac9-95cb-a629c9aac64a\") " pod="openshift-dns/dns-default-jw7lc"
Jul 13 18:50:51.956743 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:51.956612    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5a144761-c470-4ac9-95cb-a629c9aac64a-metrics-tls\") pod \"dns-default-jw7lc\" (UID: \"5a144761-c470-4ac9-95cb-a629c9aac64a\") " pod="openshift-dns/dns-default-jw7lc"
Jul 13 18:50:51.956743 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:51.956632    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a144761-c470-4ac9-95cb-a629c9aac64a-tmp-dir\") pod \"dns-default-jw7lc\" (UID: \"5a144761-c470-4ac9-95cb-a629c9aac64a\") " pod="openshift-dns/dns-default-jw7lc"
Jul 13 18:50:52.057768 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:52.057718    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqg6t\" (UniqueName: \"kubernetes.io/projected/5a144761-c470-4ac9-95cb-a629c9aac64a-kube-api-access-lqg6t\") pod \"dns-default-jw7lc\" (UID: \"5a144761-c470-4ac9-95cb-a629c9aac64a\") " pod="openshift-dns/dns-default-jw7lc"
Jul 13 18:50:52.057768 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:52.057757    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kj9k\" (UniqueName: \"kubernetes.io/projected/2ddf2ee1-c9d7-4f30-aa52-b94c6d250c70-kube-api-access-4kj9k\") pod \"ingress-canary-4rt5g\" (UID: \"2ddf2ee1-c9d7-4f30-aa52-b94c6d250c70\") " pod="openshift-ingress-canary/ingress-canary-4rt5g"
Jul 13 18:50:52.057768 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:52.057778    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5a144761-c470-4ac9-95cb-a629c9aac64a-config-volume\") pod \"dns-default-jw7lc\" (UID: \"5a144761-c470-4ac9-95cb-a629c9aac64a\") " pod="openshift-dns/dns-default-jw7lc"
Jul 13 18:50:52.058235 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:52.057811    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5a144761-c470-4ac9-95cb-a629c9aac64a-metrics-tls\") pod \"dns-default-jw7lc\" (UID: \"5a144761-c470-4ac9-95cb-a629c9aac64a\") " pod="openshift-dns/dns-default-jw7lc"
Jul 13 18:50:52.058235 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:52.057829    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a144761-c470-4ac9-95cb-a629c9aac64a-tmp-dir\") pod \"dns-default-jw7lc\" (UID: \"5a144761-c470-4ac9-95cb-a629c9aac64a\") " pod="openshift-dns/dns-default-jw7lc"
Jul 13 18:50:52.058235 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:52.057844    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2ddf2ee1-c9d7-4f30-aa52-b94c6d250c70-cert\") pod \"ingress-canary-4rt5g\" (UID: \"2ddf2ee1-c9d7-4f30-aa52-b94c6d250c70\") " pod="openshift-ingress-canary/ingress-canary-4rt5g"
Jul 13 18:50:52.058235 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:50:52.057972    2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 13 18:50:52.058235 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:50:52.058044    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a144761-c470-4ac9-95cb-a629c9aac64a-metrics-tls podName:5a144761-c470-4ac9-95cb-a629c9aac64a nodeName:}" failed. No retries permitted until 2026-07-13 18:50:52.558022095 +0000 UTC m=+39.083155507 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5a144761-c470-4ac9-95cb-a629c9aac64a-metrics-tls") pod "dns-default-jw7lc" (UID: "5a144761-c470-4ac9-95cb-a629c9aac64a") : secret "dns-default-metrics-tls" not found
Jul 13 18:50:52.058235 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:52.058220    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a144761-c470-4ac9-95cb-a629c9aac64a-tmp-dir\") pod \"dns-default-jw7lc\" (UID: \"5a144761-c470-4ac9-95cb-a629c9aac64a\") " pod="openshift-dns/dns-default-jw7lc"
Jul 13 18:50:52.058461 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:52.058443    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5a144761-c470-4ac9-95cb-a629c9aac64a-config-volume\") pod \"dns-default-jw7lc\" (UID: \"5a144761-c470-4ac9-95cb-a629c9aac64a\") " pod="openshift-dns/dns-default-jw7lc"
Jul 13 18:50:52.062817 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:52.062788    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2ddf2ee1-c9d7-4f30-aa52-b94c6d250c70-cert\") pod \"ingress-canary-4rt5g\" (UID: \"2ddf2ee1-c9d7-4f30-aa52-b94c6d250c70\") " pod="openshift-ingress-canary/ingress-canary-4rt5g"
Jul 13 18:50:52.067546 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:52.067516    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqg6t\" (UniqueName: \"kubernetes.io/projected/5a144761-c470-4ac9-95cb-a629c9aac64a-kube-api-access-lqg6t\") pod \"dns-default-jw7lc\" (UID: \"5a144761-c470-4ac9-95cb-a629c9aac64a\") " pod="openshift-dns/dns-default-jw7lc"
Jul 13 18:50:52.068029 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:52.068011    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4kj9k\" (UniqueName: \"kubernetes.io/projected/2ddf2ee1-c9d7-4f30-aa52-b94c6d250c70-kube-api-access-4kj9k\") pod \"ingress-canary-4rt5g\" (UID: \"2ddf2ee1-c9d7-4f30-aa52-b94c6d250c70\") " pod="openshift-ingress-canary/ingress-canary-4rt5g"
Jul 13 18:50:52.177414 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:52.177369    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-4rt5g"
Jul 13 18:50:52.237841 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:52.237801    2565 generic.go:356] "Generic (PLEG): container finished" podID="6ca2c2d9-346a-48c4-af31-ab3b2e66539f" containerID="41eaa89624091934dc08ee998be2b6478a4e74299e1bae091f5fdb07600f349f" exitCode=0
Jul 13 18:50:52.237994 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:52.237880    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kgrbw" event={"ID":"6ca2c2d9-346a-48c4-af31-ab3b2e66539f","Type":"ContainerDied","Data":"41eaa89624091934dc08ee998be2b6478a4e74299e1bae091f5fdb07600f349f"}
Jul 13 18:50:52.335572 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:52.335539    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-4rt5g"]
Jul 13 18:50:52.339959 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:52.339917    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ddf2ee1_c9d7_4f30_aa52_b94c6d250c70.slice/crio-e9ed6b51d63d985d0b8f18acea40bd7ee7f32cdf2013043b6ad5e12844b92b60 WatchSource:0}: Error finding container e9ed6b51d63d985d0b8f18acea40bd7ee7f32cdf2013043b6ad5e12844b92b60: Status 404 returned error can't find the container with id e9ed6b51d63d985d0b8f18acea40bd7ee7f32cdf2013043b6ad5e12844b92b60
Jul 13 18:50:52.562529 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:52.562438    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5a144761-c470-4ac9-95cb-a629c9aac64a-metrics-tls\") pod \"dns-default-jw7lc\" (UID: \"5a144761-c470-4ac9-95cb-a629c9aac64a\") " pod="openshift-dns/dns-default-jw7lc"
Jul 13 18:50:52.562673 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:50:52.562556    2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 13 18:50:52.562673 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:50:52.562613    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a144761-c470-4ac9-95cb-a629c9aac64a-metrics-tls podName:5a144761-c470-4ac9-95cb-a629c9aac64a nodeName:}" failed. No retries permitted until 2026-07-13 18:50:53.562597922 +0000 UTC m=+40.087731316 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5a144761-c470-4ac9-95cb-a629c9aac64a-metrics-tls") pod "dns-default-jw7lc" (UID: "5a144761-c470-4ac9-95cb-a629c9aac64a") : secret "dns-default-metrics-tls" not found
Jul 13 18:50:53.042472 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:53.042374    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5tgm8"
Jul 13 18:50:53.042472 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:53.042416    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qlzr5"
Jul 13 18:50:53.042472 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:53.042417    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4sl67"
Jul 13 18:50:53.045417 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:53.045390    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-dz6tv\""
Jul 13 18:50:53.048908 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:53.048663    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 13 18:50:53.048908 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:53.048759    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 13 18:50:53.048908 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:53.048772    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 13 18:50:53.048908 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:53.048795    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-gsq9l\""
Jul 13 18:50:53.048908 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:53.048663    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 13 18:50:53.242187 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:53.242148    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-4rt5g" event={"ID":"2ddf2ee1-c9d7-4f30-aa52-b94c6d250c70","Type":"ContainerStarted","Data":"e9ed6b51d63d985d0b8f18acea40bd7ee7f32cdf2013043b6ad5e12844b92b60"}
Jul 13 18:50:53.245636 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:53.245603    2565 generic.go:356] "Generic (PLEG): container finished" podID="6ca2c2d9-346a-48c4-af31-ab3b2e66539f" containerID="b0a0b77ab0e1492d18ce0ab9290e4cb6b717ca240fad73a6ff78f5450fa1fe9e" exitCode=0
Jul 13 18:50:53.245803 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:53.245660    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kgrbw" event={"ID":"6ca2c2d9-346a-48c4-af31-ab3b2e66539f","Type":"ContainerDied","Data":"b0a0b77ab0e1492d18ce0ab9290e4cb6b717ca240fad73a6ff78f5450fa1fe9e"}
Jul 13 18:50:53.572256 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:53.571970    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5a144761-c470-4ac9-95cb-a629c9aac64a-metrics-tls\") pod \"dns-default-jw7lc\" (UID: \"5a144761-c470-4ac9-95cb-a629c9aac64a\") " pod="openshift-dns/dns-default-jw7lc"
Jul 13 18:50:53.572409 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:50:53.572150    2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 13 18:50:53.572409 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:50:53.572361    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a144761-c470-4ac9-95cb-a629c9aac64a-metrics-tls podName:5a144761-c470-4ac9-95cb-a629c9aac64a nodeName:}" failed. No retries permitted until 2026-07-13 18:50:55.572332871 +0000 UTC m=+42.097466279 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5a144761-c470-4ac9-95cb-a629c9aac64a-metrics-tls") pod "dns-default-jw7lc" (UID: "5a144761-c470-4ac9-95cb-a629c9aac64a") : secret "dns-default-metrics-tls" not found
Jul 13 18:50:54.253396 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:54.253356    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kgrbw" event={"ID":"6ca2c2d9-346a-48c4-af31-ab3b2e66539f","Type":"ContainerStarted","Data":"f9db63fe1f2aaf972ad35a7c63cf2b4a25e955afdb3c6c630eb042f6a14bb506"}
Jul 13 18:50:54.298496 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:54.298429    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-kgrbw" podStartSLOduration=8.066392218 podStartE2EDuration="40.298406765s" totalImagesPullingTime="32.232014547s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 18:50:14 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:50:54.297485705 +0000 UTC m=+40.822619145" watchObservedRunningTime="2026-07-13 18:50:54.298406765 +0000 UTC m=+40.823540181"
Jul 13 18:50:54.861197 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:54.861161    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-869dcfcdcd-jfqf9"]
Jul 13 18:50:54.863477 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:54.863457    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-jfqf9"
Jul 13 18:50:54.866959 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:54.866932    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-lmk8v\""
Jul 13 18:50:54.881434 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:54.881404    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-869dcfcdcd-jfqf9"]
Jul 13 18:50:54.965772 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:54.965738    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-vds4p"]
Jul 13 18:50:54.967462 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:54.967445    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-vds4p"
Jul 13 18:50:54.977915 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:54.977878    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Jul 13 18:50:54.977915 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:54.977904    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Jul 13 18:50:54.980591 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:54.980569    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-nz6l6\""
Jul 13 18:50:54.983333 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:54.983310    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8q5fv\" (UniqueName: \"kubernetes.io/projected/06900c88-a1cf-4cdb-a7b5-c621d1d8dbbb-kube-api-access-8q5fv\") pod \"network-check-source-869dcfcdcd-jfqf9\" (UID: \"06900c88-a1cf-4cdb-a7b5-c621d1d8dbbb\") " pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-jfqf9"
Jul 13 18:50:54.989825 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:54.989798    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-vds4p"]
Jul 13 18:50:55.009483 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:55.009395    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-whnln"]
Jul 13 18:50:55.011117 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:55.011090    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-65db858894-rjnwt"]
Jul 13 18:50:55.011266 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:55.011242    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-whnln"
Jul 13 18:50:55.012835 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:55.012817    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-rjnwt"
Jul 13 18:50:55.016957 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:55.016920    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Jul 13 18:50:55.016957 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:55.016940    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 13 18:50:55.017138 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:55.017024    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Jul 13 18:50:55.017138 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:55.017094    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 13 18:50:55.018320 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:55.018301    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Jul 13 18:50:55.018409 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:55.018327    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Jul 13 18:50:55.018409 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:55.018364    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Jul 13 18:50:55.018511 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:55.018435    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-zxfcj\""
Jul 13 18:50:55.018511 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:55.018460    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-h5f5j\""
Jul 13 18:50:55.034620 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:55.034593    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-65db858894-rjnwt"]
Jul 13 18:50:55.035259 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:55.035236    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-whnln"]
Jul 13 18:50:55.071867 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:55.071832    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-7695fc6c8c-2mz4v"]
Jul 13 18:50:55.073830 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:55.073811    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-2mz4v"
Jul 13 18:50:55.080319 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:55.080293    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-rdws9\""
Jul 13 18:50:55.080427 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:55.080371    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Jul 13 18:50:55.082713 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:55.082680    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Jul 13 18:50:55.082713 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:55.082692    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Jul 13 18:50:55.083542 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:55.083522    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Jul 13 18:50:55.083648 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:55.083630    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8q5fv\" (UniqueName: \"kubernetes.io/projected/06900c88-a1cf-4cdb-a7b5-c621d1d8dbbb-kube-api-access-8q5fv\") pod \"network-check-source-869dcfcdcd-jfqf9\" (UID: \"06900c88-a1cf-4cdb-a7b5-c621d1d8dbbb\") " pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-jfqf9"
Jul 13 18:50:55.083798 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:55.083781    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5czb\" (UniqueName: \"kubernetes.io/projected/85363100-f0a6-463d-9c43-abf1539636f8-kube-api-access-p5czb\") pod \"volume-data-source-validator-78495597f6-vds4p\" (UID: \"85363100-f0a6-463d-9c43-abf1539636f8\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-vds4p"
Jul 13 18:50:55.097984 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:55.097960    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-7695fc6c8c-2mz4v"]
Jul 13 18:50:55.102879 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:55.102857    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-6cf856758c-bdj79"]
Jul 13 18:50:55.104500 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:55.104485    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6cf856758c-bdj79"
Jul 13 18:50:55.108036 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:50:55.108006    2565 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"default-dockercfg-l9grm\" is forbidden: User \"system:node:ip-10-0-128-215.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-network-console\": no relationship found between node 'ip-10-0-128-215.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-network-console\"/\"default-dockercfg-l9grm\"" type="*v1.Secret"
Jul 13 18:50:55.108036 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:50:55.108015    2565 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"networking-console-plugin-cert\" is forbidden: User \"system:node:ip-10-0-128-215.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-network-console\": no relationship found between node 'ip-10-0-128-215.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" type="*v1.Secret"
Jul 13 18:50:55.108197 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:50:55.108053    2565 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"networking-console-plugin\" is forbidden: User \"system:node:ip-10-0-128-215.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-network-console\": no relationship found between node 'ip-10-0-128-215.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" type="*v1.ConfigMap"
Jul 13 18:50:55.116342 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:55.116318    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8q5fv\" (UniqueName: \"kubernetes.io/projected/06900c88-a1cf-4cdb-a7b5-c621d1d8dbbb-kube-api-access-8q5fv\") pod \"network-check-source-869dcfcdcd-jfqf9\" (UID: \"06900c88-a1cf-4cdb-a7b5-c621d1d8dbbb\") " pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-jfqf9"
Jul 13 18:50:55.165190 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:55.165152    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6cf856758c-bdj79"]
Jul 13 18:50:55.175485 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:55.175451    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-jfqf9"
Jul 13 18:50:55.184514 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:55.184481    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1aa6cfb3-d5a7-4504-a9d6-11d63dd133f7-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-rjnwt\" (UID: \"1aa6cfb3-d5a7-4504-a9d6-11d63dd133f7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-rjnwt"
Jul 13 18:50:55.184514 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:55.184514    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6fh9\" (UniqueName: \"kubernetes.io/projected/3b153c4b-5864-4143-a644-cb49d7f31e29-kube-api-access-n6fh9\") pod \"service-ca-operator-7695fc6c8c-2mz4v\" (UID: \"3b153c4b-5864-4143-a644-cb49d7f31e29\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-2mz4v"
Jul 13 18:50:55.184834 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:55.184535    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3b153c4b-5864-4143-a644-cb49d7f31e29-serving-cert\") pod \"service-ca-operator-7695fc6c8c-2mz4v\" (UID: \"3b153c4b-5864-4143-a644-cb49d7f31e29\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-2mz4v"
Jul 13 18:50:55.184834 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:55.184581    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3b153c4b-5864-4143-a644-cb49d7f31e29-config\") pod \"service-ca-operator-7695fc6c8c-2mz4v\" (UID: \"3b153c4b-5864-4143-a644-cb49d7f31e29\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-2mz4v"
Jul 13 18:50:55.184834 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:55.184625    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ae45caa3-cc5b-4100-809e-2961c7fab65c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-whnln\" (UID: \"ae45caa3-cc5b-4100-809e-2961c7fab65c\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-whnln"
Jul 13 18:50:55.184834 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:55.184652    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6grw\" (UniqueName: \"kubernetes.io/projected/ae45caa3-cc5b-4100-809e-2961c7fab65c-kube-api-access-n6grw\") pod \"cluster-monitoring-operator-5f7f494dd4-whnln\" (UID: \"ae45caa3-cc5b-4100-809e-2961c7fab65c\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-whnln"
Jul 13 18:50:55.184834 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:55.184745    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5czb\" (UniqueName: \"kubernetes.io/projected/85363100-f0a6-463d-9c43-abf1539636f8-kube-api-access-p5czb\") pod \"volume-data-source-validator-78495597f6-vds4p\" (UID: \"85363100-f0a6-463d-9c43-abf1539636f8\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-vds4p"
Jul 13 18:50:55.184834 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:55.184770    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wb2lm\" (UniqueName: \"kubernetes.io/projected/1aa6cfb3-d5a7-4504-a9d6-11d63dd133f7-kube-api-access-wb2lm\") pod \"cluster-samples-operator-65db858894-rjnwt\" (UID: \"1aa6cfb3-d5a7-4504-a9d6-11d63dd133f7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-rjnwt"
Jul 13 18:50:55.184834 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:55.184787    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/ae45caa3-cc5b-4100-809e-2961c7fab65c-telemetry-config\") pod \"cluster-monitoring-operator-5f7f494dd4-whnln\" (UID: \"ae45caa3-cc5b-4100-809e-2961c7fab65c\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-whnln"
Jul 13 18:50:55.201412 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:55.201378    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5czb\" (UniqueName: \"kubernetes.io/projected/85363100-f0a6-463d-9c43-abf1539636f8-kube-api-access-p5czb\") pod \"volume-data-source-validator-78495597f6-vds4p\" (UID: \"85363100-f0a6-463d-9c43-abf1539636f8\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-vds4p"
Jul 13 18:50:55.263120 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:55.262878    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-4rt5g" event={"ID":"2ddf2ee1-c9d7-4f30-aa52-b94c6d250c70","Type":"ContainerStarted","Data":"715a2b422873a5f2bf9264b387caa5044dd807d1b66416eb89ab5795ae42d4fc"}
Jul 13 18:50:55.281605 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:55.281572    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-vds4p"
Jul 13 18:50:55.285521 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:55.285499    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wb2lm\" (UniqueName: \"kubernetes.io/projected/1aa6cfb3-d5a7-4504-a9d6-11d63dd133f7-kube-api-access-wb2lm\") pod \"cluster-samples-operator-65db858894-rjnwt\" (UID: \"1aa6cfb3-d5a7-4504-a9d6-11d63dd133f7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-rjnwt"
Jul 13 18:50:55.285658 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:55.285528    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/ae45caa3-cc5b-4100-809e-2961c7fab65c-telemetry-config\") pod \"cluster-monitoring-operator-5f7f494dd4-whnln\" (UID: \"ae45caa3-cc5b-4100-809e-2961c7fab65c\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-whnln"
Jul 13 18:50:55.285658 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:55.285564    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a10721d0-fc1e-485e-848b-7b877ad97903-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-bdj79\" (UID: \"a10721d0-fc1e-485e-848b-7b877ad97903\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-bdj79"
Jul 13 18:50:55.285658 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:55.285598    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1aa6cfb3-d5a7-4504-a9d6-11d63dd133f7-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-rjnwt\" (UID: \"1aa6cfb3-d5a7-4504-a9d6-11d63dd133f7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-rjnwt"
Jul 13 18:50:55.285658 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:55.285620    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6fh9\" (UniqueName: \"kubernetes.io/projected/3b153c4b-5864-4143-a644-cb49d7f31e29-kube-api-access-n6fh9\") pod \"service-ca-operator-7695fc6c8c-2mz4v\" (UID: \"3b153c4b-5864-4143-a644-cb49d7f31e29\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-2mz4v"
Jul 13 18:50:55.285889 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:50:55.285741    2565 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 13 18:50:55.285889 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:55.285806    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/a10721d0-fc1e-485e-848b-7b877ad97903-nginx-conf\") pod \"networking-console-plugin-6cf856758c-bdj79\" (UID: \"a10721d0-fc1e-485e-848b-7b877ad97903\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-bdj79"
Jul 13 18:50:55.285889 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:50:55.285828    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1aa6cfb3-d5a7-4504-a9d6-11d63dd133f7-samples-operator-tls podName:1aa6cfb3-d5a7-4504-a9d6-11d63dd133f7 nodeName:}" failed. No retries permitted until 2026-07-13 18:50:55.785805683 +0000 UTC m=+42.310939079 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/1aa6cfb3-d5a7-4504-a9d6-11d63dd133f7-samples-operator-tls") pod "cluster-samples-operator-65db858894-rjnwt" (UID: "1aa6cfb3-d5a7-4504-a9d6-11d63dd133f7") : secret "samples-operator-tls" not found
Jul 13 18:50:55.286073 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:55.285895    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3b153c4b-5864-4143-a644-cb49d7f31e29-serving-cert\") pod \"service-ca-operator-7695fc6c8c-2mz4v\" (UID: \"3b153c4b-5864-4143-a644-cb49d7f31e29\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-2mz4v"
Jul 13 18:50:55.286073 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:55.285926    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3b153c4b-5864-4143-a644-cb49d7f31e29-config\") pod \"service-ca-operator-7695fc6c8c-2mz4v\" (UID: \"3b153c4b-5864-4143-a644-cb49d7f31e29\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-2mz4v"
Jul 13 18:50:55.286073 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:55.285968    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ae45caa3-cc5b-4100-809e-2961c7fab65c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-whnln\" (UID: \"ae45caa3-cc5b-4100-809e-2961c7fab65c\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-whnln"
Jul 13 18:50:55.286073 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:55.285986    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6grw\" (UniqueName: \"kubernetes.io/projected/ae45caa3-cc5b-4100-809e-2961c7fab65c-kube-api-access-n6grw\") pod \"cluster-monitoring-operator-5f7f494dd4-whnln\" (UID: \"ae45caa3-cc5b-4100-809e-2961c7fab65c\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-whnln"
Jul 13 18:50:55.286303 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:50:55.286249    2565 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 13 18:50:55.286303 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:50:55.286295    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ae45caa3-cc5b-4100-809e-2961c7fab65c-cluster-monitoring-operator-tls podName:ae45caa3-cc5b-4100-809e-2961c7fab65c nodeName:}" failed. No retries permitted until 2026-07-13 18:50:55.786282604 +0000 UTC m=+42.311415998 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/ae45caa3-cc5b-4100-809e-2961c7fab65c-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-whnln" (UID: "ae45caa3-cc5b-4100-809e-2961c7fab65c") : secret "cluster-monitoring-operator-tls" not found
Jul 13 18:50:55.286459 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:55.286437    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/ae45caa3-cc5b-4100-809e-2961c7fab65c-telemetry-config\") pod \"cluster-monitoring-operator-5f7f494dd4-whnln\" (UID: \"ae45caa3-cc5b-4100-809e-2961c7fab65c\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-whnln"
Jul 13 18:50:55.286541 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:55.286522    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3b153c4b-5864-4143-a644-cb49d7f31e29-config\") pod \"service-ca-operator-7695fc6c8c-2mz4v\" (UID: \"3b153c4b-5864-4143-a644-cb49d7f31e29\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-2mz4v"
Jul 13 18:50:55.289163 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:55.289133    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3b153c4b-5864-4143-a644-cb49d7f31e29-serving-cert\") pod \"service-ca-operator-7695fc6c8c-2mz4v\" (UID: \"3b153c4b-5864-4143-a644-cb49d7f31e29\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-2mz4v"
Jul 13 18:50:55.300660 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:55.300612    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-4rt5g" podStartSLOduration=1.881938929 podStartE2EDuration="4.300595193s" totalImagesPullingTime="2.418656264s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 18:50:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:50:55.300473257 +0000 UTC m=+41.825606694" watchObservedRunningTime="2026-07-13 18:50:55.300595193 +0000 UTC m=+41.825728600"
Jul 13 18:50:55.316868 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:55.316836    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6fh9\" (UniqueName: \"kubernetes.io/projected/3b153c4b-5864-4143-a644-cb49d7f31e29-kube-api-access-n6fh9\") pod \"service-ca-operator-7695fc6c8c-2mz4v\" (UID: \"3b153c4b-5864-4143-a644-cb49d7f31e29\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-2mz4v"
Jul 13 18:50:55.317267 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:55.317241    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wb2lm\" (UniqueName: \"kubernetes.io/projected/1aa6cfb3-d5a7-4504-a9d6-11d63dd133f7-kube-api-access-wb2lm\") pod \"cluster-samples-operator-65db858894-rjnwt\" (UID: \"1aa6cfb3-d5a7-4504-a9d6-11d63dd133f7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-rjnwt"
Jul 13 18:50:55.322492 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:55.322455    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6grw\" (UniqueName: \"kubernetes.io/projected/ae45caa3-cc5b-4100-809e-2961c7fab65c-kube-api-access-n6grw\") pod \"cluster-monitoring-operator-5f7f494dd4-whnln\" (UID: \"ae45caa3-cc5b-4100-809e-2961c7fab65c\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-whnln"
Jul 13 18:50:55.334982 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:55.334946    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-869dcfcdcd-jfqf9"]
Jul 13 18:50:55.339228 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:55.339187    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06900c88_a1cf_4cdb_a7b5_c621d1d8dbbb.slice/crio-c6e4448517e3e168eb3c3bcd846d5169b143e512e3a9ba7016c80f989dfbd7a6 WatchSource:0}: Error finding container c6e4448517e3e168eb3c3bcd846d5169b143e512e3a9ba7016c80f989dfbd7a6: Status 404 returned error can't find the container with id c6e4448517e3e168eb3c3bcd846d5169b143e512e3a9ba7016c80f989dfbd7a6
Jul 13 18:50:55.381994 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:55.381964    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-2mz4v"
Jul 13 18:50:55.387033 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:55.386996    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/a10721d0-fc1e-485e-848b-7b877ad97903-nginx-conf\") pod \"networking-console-plugin-6cf856758c-bdj79\" (UID: \"a10721d0-fc1e-485e-848b-7b877ad97903\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-bdj79"
Jul 13 18:50:55.387736 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:55.387176    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a10721d0-fc1e-485e-848b-7b877ad97903-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-bdj79\" (UID: \"a10721d0-fc1e-485e-848b-7b877ad97903\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-bdj79"
Jul 13 18:50:55.430732 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:55.430558    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-vds4p"]
Jul 13 18:50:55.435317 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:55.435276    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85363100_f0a6_463d_9c43_abf1539636f8.slice/crio-96bdf441f435396d677209a04b34a7f58a765c8ae077be0cce3c035f2eb916f7 WatchSource:0}: Error finding container 96bdf441f435396d677209a04b34a7f58a765c8ae077be0cce3c035f2eb916f7: Status 404 returned error can't find the container with id 96bdf441f435396d677209a04b34a7f58a765c8ae077be0cce3c035f2eb916f7
Jul 13 18:50:55.514103 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:55.514024    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-7695fc6c8c-2mz4v"]
Jul 13 18:50:55.517463 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:50:55.517437    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b153c4b_5864_4143_a644_cb49d7f31e29.slice/crio-3dff67c1df70ff4e1976d84fdaf57af98477981c2b3b8665a7944167ed05adc9 WatchSource:0}: Error finding container 3dff67c1df70ff4e1976d84fdaf57af98477981c2b3b8665a7944167ed05adc9: Status 404 returned error can't find the container with id 3dff67c1df70ff4e1976d84fdaf57af98477981c2b3b8665a7944167ed05adc9
Jul 13 18:50:55.588329 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:55.588292    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5a144761-c470-4ac9-95cb-a629c9aac64a-metrics-tls\") pod \"dns-default-jw7lc\" (UID: \"5a144761-c470-4ac9-95cb-a629c9aac64a\") " pod="openshift-dns/dns-default-jw7lc"
Jul 13 18:50:55.589243 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:50:55.588843    2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 13 18:50:55.589243 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:50:55.588948    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a144761-c470-4ac9-95cb-a629c9aac64a-metrics-tls podName:5a144761-c470-4ac9-95cb-a629c9aac64a nodeName:}" failed. No retries permitted until 2026-07-13 18:50:59.588927173 +0000 UTC m=+46.114060584 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5a144761-c470-4ac9-95cb-a629c9aac64a-metrics-tls") pod "dns-default-jw7lc" (UID: "5a144761-c470-4ac9-95cb-a629c9aac64a") : secret "dns-default-metrics-tls" not found
Jul 13 18:50:55.790123 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:55.790030    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ae45caa3-cc5b-4100-809e-2961c7fab65c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-whnln\" (UID: \"ae45caa3-cc5b-4100-809e-2961c7fab65c\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-whnln"
Jul 13 18:50:55.790123 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:55.790119    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1aa6cfb3-d5a7-4504-a9d6-11d63dd133f7-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-rjnwt\" (UID: \"1aa6cfb3-d5a7-4504-a9d6-11d63dd133f7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-rjnwt"
Jul 13 18:50:55.790298 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:50:55.790176    2565 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 13 18:50:55.790298 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:50:55.790214    2565 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 13 18:50:55.790298 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:50:55.790243    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ae45caa3-cc5b-4100-809e-2961c7fab65c-cluster-monitoring-operator-tls podName:ae45caa3-cc5b-4100-809e-2961c7fab65c nodeName:}" failed. No retries permitted until 2026-07-13 18:50:56.790226814 +0000 UTC m=+43.315360212 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/ae45caa3-cc5b-4100-809e-2961c7fab65c-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-whnln" (UID: "ae45caa3-cc5b-4100-809e-2961c7fab65c") : secret "cluster-monitoring-operator-tls" not found
Jul 13 18:50:55.790298 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:50:55.790259    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1aa6cfb3-d5a7-4504-a9d6-11d63dd133f7-samples-operator-tls podName:1aa6cfb3-d5a7-4504-a9d6-11d63dd133f7 nodeName:}" failed. No retries permitted until 2026-07-13 18:50:56.790252389 +0000 UTC m=+43.315385782 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/1aa6cfb3-d5a7-4504-a9d6-11d63dd133f7-samples-operator-tls") pod "cluster-samples-operator-65db858894-rjnwt" (UID: "1aa6cfb3-d5a7-4504-a9d6-11d63dd133f7") : secret "samples-operator-tls" not found
Jul 13 18:50:56.268200 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:56.268113    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-jfqf9" event={"ID":"06900c88-a1cf-4cdb-a7b5-c621d1d8dbbb","Type":"ContainerStarted","Data":"c6e4448517e3e168eb3c3bcd846d5169b143e512e3a9ba7016c80f989dfbd7a6"}
Jul 13 18:50:56.269987 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:56.269945    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-2mz4v" event={"ID":"3b153c4b-5864-4143-a644-cb49d7f31e29","Type":"ContainerStarted","Data":"3dff67c1df70ff4e1976d84fdaf57af98477981c2b3b8665a7944167ed05adc9"}
Jul 13 18:50:56.271614 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:56.271580    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-vds4p" event={"ID":"85363100-f0a6-463d-9c43-abf1539636f8","Type":"ContainerStarted","Data":"96bdf441f435396d677209a04b34a7f58a765c8ae077be0cce3c035f2eb916f7"}
Jul 13 18:50:56.307643 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:56.307610    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-l9grm\""
Jul 13 18:50:56.387931 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:50:56.387566    2565 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: failed to sync secret cache: timed out waiting for the condition
Jul 13 18:50:56.387931 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:50:56.387666    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a10721d0-fc1e-485e-848b-7b877ad97903-networking-console-plugin-cert podName:a10721d0-fc1e-485e-848b-7b877ad97903 nodeName:}" failed. No retries permitted until 2026-07-13 18:50:56.887641571 +0000 UTC m=+43.412774974 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/a10721d0-fc1e-485e-848b-7b877ad97903-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-bdj79" (UID: "a10721d0-fc1e-485e-848b-7b877ad97903") : failed to sync secret cache: timed out waiting for the condition
Jul 13 18:50:56.387931 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:50:56.387576    2565 configmap.go:193] Couldn't get configMap openshift-network-console/networking-console-plugin: failed to sync configmap cache: timed out waiting for the condition
Jul 13 18:50:56.387931 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:50:56.387771    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a10721d0-fc1e-485e-848b-7b877ad97903-nginx-conf podName:a10721d0-fc1e-485e-848b-7b877ad97903 nodeName:}" failed. No retries permitted until 2026-07-13 18:50:56.887756147 +0000 UTC m=+43.412889542 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "nginx-conf" (UniqueName: "kubernetes.io/configmap/a10721d0-fc1e-485e-848b-7b877ad97903-nginx-conf") pod "networking-console-plugin-6cf856758c-bdj79" (UID: "a10721d0-fc1e-485e-848b-7b877ad97903") : failed to sync configmap cache: timed out waiting for the condition
Jul 13 18:50:56.584489 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:56.584396    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Jul 13 18:50:56.657762 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:56.657726    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Jul 13 18:50:56.798607 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:56.798566    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1aa6cfb3-d5a7-4504-a9d6-11d63dd133f7-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-rjnwt\" (UID: \"1aa6cfb3-d5a7-4504-a9d6-11d63dd133f7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-rjnwt"
Jul 13 18:50:56.798777 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:56.798686    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ae45caa3-cc5b-4100-809e-2961c7fab65c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-whnln\" (UID: \"ae45caa3-cc5b-4100-809e-2961c7fab65c\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-whnln"
Jul 13 18:50:56.798777 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:50:56.798750    2565 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 13 18:50:56.798951 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:50:56.798826    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1aa6cfb3-d5a7-4504-a9d6-11d63dd133f7-samples-operator-tls podName:1aa6cfb3-d5a7-4504-a9d6-11d63dd133f7 nodeName:}" failed. No retries permitted until 2026-07-13 18:50:58.798804492 +0000 UTC m=+45.323937921 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/1aa6cfb3-d5a7-4504-a9d6-11d63dd133f7-samples-operator-tls") pod "cluster-samples-operator-65db858894-rjnwt" (UID: "1aa6cfb3-d5a7-4504-a9d6-11d63dd133f7") : secret "samples-operator-tls" not found
Jul 13 18:50:56.798951 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:50:56.798843    2565 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 13 18:50:56.798951 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:50:56.798906    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ae45caa3-cc5b-4100-809e-2961c7fab65c-cluster-monitoring-operator-tls podName:ae45caa3-cc5b-4100-809e-2961c7fab65c nodeName:}" failed. No retries permitted until 2026-07-13 18:50:58.798886123 +0000 UTC m=+45.324019520 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/ae45caa3-cc5b-4100-809e-2961c7fab65c-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-whnln" (UID: "ae45caa3-cc5b-4100-809e-2961c7fab65c") : secret "cluster-monitoring-operator-tls" not found
Jul 13 18:50:56.900117 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:56.900020    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a10721d0-fc1e-485e-848b-7b877ad97903-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-bdj79\" (UID: \"a10721d0-fc1e-485e-848b-7b877ad97903\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-bdj79"
Jul 13 18:50:56.900117 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:56.900096    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/a10721d0-fc1e-485e-848b-7b877ad97903-nginx-conf\") pod \"networking-console-plugin-6cf856758c-bdj79\" (UID: \"a10721d0-fc1e-485e-848b-7b877ad97903\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-bdj79"
Jul 13 18:50:56.900303 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:50:56.900180    2565 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 13 18:50:56.900303 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:50:56.900252    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a10721d0-fc1e-485e-848b-7b877ad97903-networking-console-plugin-cert podName:a10721d0-fc1e-485e-848b-7b877ad97903 nodeName:}" failed. No retries permitted until 2026-07-13 18:50:57.900236393 +0000 UTC m=+44.425369792 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/a10721d0-fc1e-485e-848b-7b877ad97903-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-bdj79" (UID: "a10721d0-fc1e-485e-848b-7b877ad97903") : secret "networking-console-plugin-cert" not found
Jul 13 18:50:56.900735 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:56.900695    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/a10721d0-fc1e-485e-848b-7b877ad97903-nginx-conf\") pod \"networking-console-plugin-6cf856758c-bdj79\" (UID: \"a10721d0-fc1e-485e-848b-7b877ad97903\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-bdj79"
Jul 13 18:50:57.277506 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:57.277450    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-vds4p" event={"ID":"85363100-f0a6-463d-9c43-abf1539636f8","Type":"ContainerStarted","Data":"e29af9d23d84d598717d059b019fa515d368c43e7bd75a54d34543854a5b251f"}
Jul 13 18:50:57.294901 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:57.294834    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-vds4p" podStartSLOduration=1.964132329 podStartE2EDuration="3.294812236s" totalImagesPullingTime="1.330679907s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 18:50:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:50:57.293185206 +0000 UTC m=+43.818318644" watchObservedRunningTime="2026-07-13 18:50:57.294812236 +0000 UTC m=+43.819945654"
Jul 13 18:50:57.906837 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:57.906789    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a10721d0-fc1e-485e-848b-7b877ad97903-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-bdj79\" (UID: \"a10721d0-fc1e-485e-848b-7b877ad97903\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-bdj79"
Jul 13 18:50:57.907073 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:50:57.907051    2565 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 13 18:50:57.907167 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:50:57.907130    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a10721d0-fc1e-485e-848b-7b877ad97903-networking-console-plugin-cert podName:a10721d0-fc1e-485e-848b-7b877ad97903 nodeName:}" failed. No retries permitted until 2026-07-13 18:50:59.907108923 +0000 UTC m=+46.432242339 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/a10721d0-fc1e-485e-848b-7b877ad97903-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-bdj79" (UID: "a10721d0-fc1e-485e-848b-7b877ad97903") : secret "networking-console-plugin-cert" not found
Jul 13 18:50:58.813860 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:58.813809    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ae45caa3-cc5b-4100-809e-2961c7fab65c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-whnln\" (UID: \"ae45caa3-cc5b-4100-809e-2961c7fab65c\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-whnln"
Jul 13 18:50:58.814327 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:58.813909    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1aa6cfb3-d5a7-4504-a9d6-11d63dd133f7-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-rjnwt\" (UID: \"1aa6cfb3-d5a7-4504-a9d6-11d63dd133f7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-rjnwt"
Jul 13 18:50:58.814327 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:50:58.813952    2565 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 13 18:50:58.814327 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:50:58.814011    2565 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 13 18:50:58.814327 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:50:58.814022    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ae45caa3-cc5b-4100-809e-2961c7fab65c-cluster-monitoring-operator-tls podName:ae45caa3-cc5b-4100-809e-2961c7fab65c nodeName:}" failed. No retries permitted until 2026-07-13 18:51:02.814005331 +0000 UTC m=+49.339138724 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/ae45caa3-cc5b-4100-809e-2961c7fab65c-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-whnln" (UID: "ae45caa3-cc5b-4100-809e-2961c7fab65c") : secret "cluster-monitoring-operator-tls" not found
Jul 13 18:50:58.814327 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:50:58.814052    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1aa6cfb3-d5a7-4504-a9d6-11d63dd133f7-samples-operator-tls podName:1aa6cfb3-d5a7-4504-a9d6-11d63dd133f7 nodeName:}" failed. No retries permitted until 2026-07-13 18:51:02.814041461 +0000 UTC m=+49.339174854 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/1aa6cfb3-d5a7-4504-a9d6-11d63dd133f7-samples-operator-tls") pod "cluster-samples-operator-65db858894-rjnwt" (UID: "1aa6cfb3-d5a7-4504-a9d6-11d63dd133f7") : secret "samples-operator-tls" not found
Jul 13 18:50:59.286188 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:59.285766    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-2mz4v" event={"ID":"3b153c4b-5864-4143-a644-cb49d7f31e29","Type":"ContainerStarted","Data":"0a9c642bac369bfb4709307fbf3700de0699c1cc76e6a7c2d2ce9d71206ef1e7"}
Jul 13 18:50:59.304192 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:59.304128    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-2mz4v" podStartSLOduration=1.439223237 podStartE2EDuration="4.304103867s" totalImagesPullingTime="2.86488063s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 18:50:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:50:59.3031332 +0000 UTC m=+45.828266636" watchObservedRunningTime="2026-07-13 18:50:59.304103867 +0000 UTC m=+45.829237278"
Jul 13 18:50:59.620231 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:59.620138    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5a144761-c470-4ac9-95cb-a629c9aac64a-metrics-tls\") pod \"dns-default-jw7lc\" (UID: \"5a144761-c470-4ac9-95cb-a629c9aac64a\") " pod="openshift-dns/dns-default-jw7lc"
Jul 13 18:50:59.620418 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:50:59.620315    2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 13 18:50:59.620418 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:50:59.620400    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a144761-c470-4ac9-95cb-a629c9aac64a-metrics-tls podName:5a144761-c470-4ac9-95cb-a629c9aac64a nodeName:}" failed. No retries permitted until 2026-07-13 18:51:07.62037876 +0000 UTC m=+54.145512154 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5a144761-c470-4ac9-95cb-a629c9aac64a-metrics-tls") pod "dns-default-jw7lc" (UID: "5a144761-c470-4ac9-95cb-a629c9aac64a") : secret "dns-default-metrics-tls" not found
Jul 13 18:50:59.728270 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:59.728233    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-xmzjq"]
Jul 13 18:50:59.757120 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:59.756064    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-xmzjq"]
Jul 13 18:50:59.757120 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:59.756211    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-xmzjq"
Jul 13 18:50:59.759304 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:59.759227    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 13 18:50:59.759915 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:59.759608    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 13 18:50:59.760231 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:59.760207    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-c5j2r\""
Jul 13 18:50:59.760457 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:59.760441    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 13 18:50:59.763511 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:59.763492    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 13 18:50:59.821609 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:59.821566    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbs4h\" (UniqueName: \"kubernetes.io/projected/053446da-3c51-46b2-a759-aad264c7b0f9-kube-api-access-hbs4h\") pod \"insights-runtime-extractor-xmzjq\" (UID: \"053446da-3c51-46b2-a759-aad264c7b0f9\") " pod="openshift-insights/insights-runtime-extractor-xmzjq"
Jul 13 18:50:59.822016 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:59.821615    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/053446da-3c51-46b2-a759-aad264c7b0f9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xmzjq\" (UID: \"053446da-3c51-46b2-a759-aad264c7b0f9\") " pod="openshift-insights/insights-runtime-extractor-xmzjq"
Jul 13 18:50:59.822016 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:59.821726    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/053446da-3c51-46b2-a759-aad264c7b0f9-data-volume\") pod \"insights-runtime-extractor-xmzjq\" (UID: \"053446da-3c51-46b2-a759-aad264c7b0f9\") " pod="openshift-insights/insights-runtime-extractor-xmzjq"
Jul 13 18:50:59.822016 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:59.821832    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/053446da-3c51-46b2-a759-aad264c7b0f9-crio-socket\") pod \"insights-runtime-extractor-xmzjq\" (UID: \"053446da-3c51-46b2-a759-aad264c7b0f9\") " pod="openshift-insights/insights-runtime-extractor-xmzjq"
Jul 13 18:50:59.822016 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:59.821851    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/053446da-3c51-46b2-a759-aad264c7b0f9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xmzjq\" (UID: \"053446da-3c51-46b2-a759-aad264c7b0f9\") " pod="openshift-insights/insights-runtime-extractor-xmzjq"
Jul 13 18:50:59.922853 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:59.922758    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/053446da-3c51-46b2-a759-aad264c7b0f9-data-volume\") pod \"insights-runtime-extractor-xmzjq\" (UID: \"053446da-3c51-46b2-a759-aad264c7b0f9\") " pod="openshift-insights/insights-runtime-extractor-xmzjq"
Jul 13 18:50:59.923000 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:59.922892    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/053446da-3c51-46b2-a759-aad264c7b0f9-crio-socket\") pod \"insights-runtime-extractor-xmzjq\" (UID: \"053446da-3c51-46b2-a759-aad264c7b0f9\") " pod="openshift-insights/insights-runtime-extractor-xmzjq"
Jul 13 18:50:59.923000 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:59.922919    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/053446da-3c51-46b2-a759-aad264c7b0f9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xmzjq\" (UID: \"053446da-3c51-46b2-a759-aad264c7b0f9\") " pod="openshift-insights/insights-runtime-extractor-xmzjq"
Jul 13 18:50:59.923000 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:59.922967    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a10721d0-fc1e-485e-848b-7b877ad97903-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-bdj79\" (UID: \"a10721d0-fc1e-485e-848b-7b877ad97903\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-bdj79"
Jul 13 18:50:59.923143 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:50:59.923057    2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 13 18:50:59.923143 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:50:59.923073    2565 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 13 18:50:59.923143 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:59.923103    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbs4h\" (UniqueName: \"kubernetes.io/projected/053446da-3c51-46b2-a759-aad264c7b0f9-kube-api-access-hbs4h\") pod \"insights-runtime-extractor-xmzjq\" (UID: \"053446da-3c51-46b2-a759-aad264c7b0f9\") " pod="openshift-insights/insights-runtime-extractor-xmzjq"
Jul 13 18:50:59.923143 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:50:59.923136    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/053446da-3c51-46b2-a759-aad264c7b0f9-kube-rbac-proxy-cm podName:053446da-3c51-46b2-a759-aad264c7b0f9 nodeName:}" failed. No retries permitted until 2026-07-13 18:51:00.423118982 +0000 UTC m=+46.948252375 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/053446da-3c51-46b2-a759-aad264c7b0f9-kube-rbac-proxy-cm") pod "insights-runtime-extractor-xmzjq" (UID: "053446da-3c51-46b2-a759-aad264c7b0f9") : configmap "kube-rbac-proxy" not found
Jul 13 18:50:59.923293 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:59.923152    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/053446da-3c51-46b2-a759-aad264c7b0f9-data-volume\") pod \"insights-runtime-extractor-xmzjq\" (UID: \"053446da-3c51-46b2-a759-aad264c7b0f9\") " pod="openshift-insights/insights-runtime-extractor-xmzjq"
Jul 13 18:50:59.923293 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:59.923153    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/053446da-3c51-46b2-a759-aad264c7b0f9-crio-socket\") pod \"insights-runtime-extractor-xmzjq\" (UID: \"053446da-3c51-46b2-a759-aad264c7b0f9\") " pod="openshift-insights/insights-runtime-extractor-xmzjq"
Jul 13 18:50:59.923293 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:50:59.923186    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a10721d0-fc1e-485e-848b-7b877ad97903-networking-console-plugin-cert podName:a10721d0-fc1e-485e-848b-7b877ad97903 nodeName:}" failed. No retries permitted until 2026-07-13 18:51:03.923161685 +0000 UTC m=+50.448295081 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/a10721d0-fc1e-485e-848b-7b877ad97903-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-bdj79" (UID: "a10721d0-fc1e-485e-848b-7b877ad97903") : secret "networking-console-plugin-cert" not found
Jul 13 18:50:59.923293 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:59.923241    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/053446da-3c51-46b2-a759-aad264c7b0f9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xmzjq\" (UID: \"053446da-3c51-46b2-a759-aad264c7b0f9\") " pod="openshift-insights/insights-runtime-extractor-xmzjq"
Jul 13 18:50:59.923432 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:50:59.923346    2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 13 18:50:59.923432 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:50:59.923394    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/053446da-3c51-46b2-a759-aad264c7b0f9-insights-runtime-extractor-tls podName:053446da-3c51-46b2-a759-aad264c7b0f9 nodeName:}" failed. No retries permitted until 2026-07-13 18:51:00.423369208 +0000 UTC m=+46.948502603 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/053446da-3c51-46b2-a759-aad264c7b0f9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xmzjq" (UID: "053446da-3c51-46b2-a759-aad264c7b0f9") : secret "insights-runtime-extractor-tls" not found
Jul 13 18:50:59.934093 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:50:59.934063    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbs4h\" (UniqueName: \"kubernetes.io/projected/053446da-3c51-46b2-a759-aad264c7b0f9-kube-api-access-hbs4h\") pod \"insights-runtime-extractor-xmzjq\" (UID: \"053446da-3c51-46b2-a759-aad264c7b0f9\") " pod="openshift-insights/insights-runtime-extractor-xmzjq"
Jul 13 18:51:00.426806 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:00.426755    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/053446da-3c51-46b2-a759-aad264c7b0f9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xmzjq\" (UID: \"053446da-3c51-46b2-a759-aad264c7b0f9\") " pod="openshift-insights/insights-runtime-extractor-xmzjq"
Jul 13 18:51:00.427012 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:00.426835    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/053446da-3c51-46b2-a759-aad264c7b0f9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xmzjq\" (UID: \"053446da-3c51-46b2-a759-aad264c7b0f9\") " pod="openshift-insights/insights-runtime-extractor-xmzjq"
Jul 13 18:51:00.427012 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:51:00.426872    2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 13 18:51:00.427012 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:51:00.426938    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/053446da-3c51-46b2-a759-aad264c7b0f9-kube-rbac-proxy-cm podName:053446da-3c51-46b2-a759-aad264c7b0f9 nodeName:}" failed. No retries permitted until 2026-07-13 18:51:01.426922102 +0000 UTC m=+47.952055497 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/053446da-3c51-46b2-a759-aad264c7b0f9-kube-rbac-proxy-cm") pod "insights-runtime-extractor-xmzjq" (UID: "053446da-3c51-46b2-a759-aad264c7b0f9") : configmap "kube-rbac-proxy" not found
Jul 13 18:51:00.427012 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:51:00.426988    2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 13 18:51:00.427218 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:51:00.427058    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/053446da-3c51-46b2-a759-aad264c7b0f9-insights-runtime-extractor-tls podName:053446da-3c51-46b2-a759-aad264c7b0f9 nodeName:}" failed. No retries permitted until 2026-07-13 18:51:01.427045638 +0000 UTC m=+47.952179033 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/053446da-3c51-46b2-a759-aad264c7b0f9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xmzjq" (UID: "053446da-3c51-46b2-a759-aad264c7b0f9") : secret "insights-runtime-extractor-tls" not found
Jul 13 18:51:01.045418 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:01.045385    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7f59b54f6c-bjfpf"]
Jul 13 18:51:01.063817 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:01.063779    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7f59b54f6c-bjfpf"]
Jul 13 18:51:01.064057 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:01.063893    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-bjfpf"
Jul 13 18:51:01.066781 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:01.066759    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-dst7k\""
Jul 13 18:51:01.067614 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:01.067596    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Jul 13 18:51:01.067719 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:01.067601    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Jul 13 18:51:01.131297 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:01.131258    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbwwj\" (UniqueName: \"kubernetes.io/projected/f4e3f129-6df0-4398-ad91-634ac530dba1-kube-api-access-hbwwj\") pod \"migrator-7f59b54f6c-bjfpf\" (UID: \"f4e3f129-6df0-4398-ad91-634ac530dba1\") " pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-bjfpf"
Jul 13 18:51:01.232349 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:01.232315    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbwwj\" (UniqueName: \"kubernetes.io/projected/f4e3f129-6df0-4398-ad91-634ac530dba1-kube-api-access-hbwwj\") pod \"migrator-7f59b54f6c-bjfpf\" (UID: \"f4e3f129-6df0-4398-ad91-634ac530dba1\") " pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-bjfpf"
Jul 13 18:51:01.242388 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:01.242348    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbwwj\" (UniqueName: \"kubernetes.io/projected/f4e3f129-6df0-4398-ad91-634ac530dba1-kube-api-access-hbwwj\") pod \"migrator-7f59b54f6c-bjfpf\" (UID: \"f4e3f129-6df0-4398-ad91-634ac530dba1\") " pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-bjfpf"
Jul 13 18:51:01.292335 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:01.292292    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-jfqf9" event={"ID":"06900c88-a1cf-4cdb-a7b5-c621d1d8dbbb","Type":"ContainerStarted","Data":"a82e0eccbd517a62f5ae4ce1e64937e23e42f7ddee56bf67728e45f097b56a99"}
Jul 13 18:51:01.313532 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:01.313428    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-jfqf9" podStartSLOduration=2.392082581 podStartE2EDuration="7.313410996s" totalImagesPullingTime="4.921328415s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 18:50:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:51:01.311799089 +0000 UTC m=+47.836932525" watchObservedRunningTime="2026-07-13 18:51:01.313410996 +0000 UTC m=+47.838544411"
Jul 13 18:51:01.373236 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:01.373198    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-bjfpf"
Jul 13 18:51:01.434541 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:01.434506    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/053446da-3c51-46b2-a759-aad264c7b0f9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xmzjq\" (UID: \"053446da-3c51-46b2-a759-aad264c7b0f9\") " pod="openshift-insights/insights-runtime-extractor-xmzjq"
Jul 13 18:51:01.434759 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:01.434566    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/053446da-3c51-46b2-a759-aad264c7b0f9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xmzjq\" (UID: \"053446da-3c51-46b2-a759-aad264c7b0f9\") " pod="openshift-insights/insights-runtime-extractor-xmzjq"
Jul 13 18:51:01.434759 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:51:01.434677    2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 13 18:51:01.434759 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:51:01.434676    2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 13 18:51:01.434759 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:51:01.434761    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/053446da-3c51-46b2-a759-aad264c7b0f9-insights-runtime-extractor-tls podName:053446da-3c51-46b2-a759-aad264c7b0f9 nodeName:}" failed. No retries permitted until 2026-07-13 18:51:03.434746045 +0000 UTC m=+49.959879439 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/053446da-3c51-46b2-a759-aad264c7b0f9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xmzjq" (UID: "053446da-3c51-46b2-a759-aad264c7b0f9") : secret "insights-runtime-extractor-tls" not found
Jul 13 18:51:01.434923 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:51:01.434776    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/053446da-3c51-46b2-a759-aad264c7b0f9-kube-rbac-proxy-cm podName:053446da-3c51-46b2-a759-aad264c7b0f9 nodeName:}" failed. No retries permitted until 2026-07-13 18:51:03.43477001 +0000 UTC m=+49.959903405 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/053446da-3c51-46b2-a759-aad264c7b0f9-kube-rbac-proxy-cm") pod "insights-runtime-extractor-xmzjq" (UID: "053446da-3c51-46b2-a759-aad264c7b0f9") : configmap "kube-rbac-proxy" not found
Jul 13 18:51:01.563831 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:01.563728    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7f59b54f6c-bjfpf"]
Jul 13 18:51:01.869147 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:01.869064    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-zcq5p_6ae58b1e-3aaf-45dc-a741-2fb895468ca7/dns-node-resolver/0.log"
Jul 13 18:51:02.296675 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:02.296634    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-bjfpf" event={"ID":"f4e3f129-6df0-4398-ad91-634ac530dba1","Type":"ContainerStarted","Data":"d7c023adda3367e964acac5096410b0ca3dcdfe3a3eccaedfb0bf36ccf2b69f6"}
Jul 13 18:51:02.588144 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:02.588069    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-79f68b9f4f-r6xjw"]
Jul 13 18:51:02.604230 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:02.604182    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-79f68b9f4f-r6xjw"]
Jul 13 18:51:02.604407 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:02.604299    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-79f68b9f4f-r6xjw"
Jul 13 18:51:02.607057 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:02.607030    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Jul 13 18:51:02.607269 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:02.607100    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Jul 13 18:51:02.607347 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:02.607334    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Jul 13 18:51:02.608337 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:02.608316    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Jul 13 18:51:02.608454 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:02.608408    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-km485\""
Jul 13 18:51:02.645037 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:02.644996    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/f49e0954-6f1a-4f69-888c-662c5dd14c14-signing-cabundle\") pod \"service-ca-79f68b9f4f-r6xjw\" (UID: \"f49e0954-6f1a-4f69-888c-662c5dd14c14\") " pod="openshift-service-ca/service-ca-79f68b9f4f-r6xjw"
Jul 13 18:51:02.645186 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:02.645116    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/f49e0954-6f1a-4f69-888c-662c5dd14c14-signing-key\") pod \"service-ca-79f68b9f4f-r6xjw\" (UID: \"f49e0954-6f1a-4f69-888c-662c5dd14c14\") " pod="openshift-service-ca/service-ca-79f68b9f4f-r6xjw"
Jul 13 18:51:02.645231 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:02.645178    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57cd4\" (UniqueName: \"kubernetes.io/projected/f49e0954-6f1a-4f69-888c-662c5dd14c14-kube-api-access-57cd4\") pod \"service-ca-79f68b9f4f-r6xjw\" (UID: \"f49e0954-6f1a-4f69-888c-662c5dd14c14\") " pod="openshift-service-ca/service-ca-79f68b9f4f-r6xjw"
Jul 13 18:51:02.669986 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:02.669953    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-k2vg4_b7240a5a-c704-49a7-b152-bdeb36cee9d4/node-ca/0.log"
Jul 13 18:51:02.745550 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:02.745514    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/f49e0954-6f1a-4f69-888c-662c5dd14c14-signing-key\") pod \"service-ca-79f68b9f4f-r6xjw\" (UID: \"f49e0954-6f1a-4f69-888c-662c5dd14c14\") " pod="openshift-service-ca/service-ca-79f68b9f4f-r6xjw"
Jul 13 18:51:02.745782 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:02.745570    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57cd4\" (UniqueName: \"kubernetes.io/projected/f49e0954-6f1a-4f69-888c-662c5dd14c14-kube-api-access-57cd4\") pod \"service-ca-79f68b9f4f-r6xjw\" (UID: \"f49e0954-6f1a-4f69-888c-662c5dd14c14\") " pod="openshift-service-ca/service-ca-79f68b9f4f-r6xjw"
Jul 13 18:51:02.745782 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:02.745635    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/f49e0954-6f1a-4f69-888c-662c5dd14c14-signing-cabundle\") pod \"service-ca-79f68b9f4f-r6xjw\" (UID: \"f49e0954-6f1a-4f69-888c-662c5dd14c14\") " pod="openshift-service-ca/service-ca-79f68b9f4f-r6xjw"
Jul 13 18:51:02.746508 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:02.746461    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/f49e0954-6f1a-4f69-888c-662c5dd14c14-signing-cabundle\") pod \"service-ca-79f68b9f4f-r6xjw\" (UID: \"f49e0954-6f1a-4f69-888c-662c5dd14c14\") " pod="openshift-service-ca/service-ca-79f68b9f4f-r6xjw"
Jul 13 18:51:02.748492 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:02.748465    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/f49e0954-6f1a-4f69-888c-662c5dd14c14-signing-key\") pod \"service-ca-79f68b9f4f-r6xjw\" (UID: \"f49e0954-6f1a-4f69-888c-662c5dd14c14\") " pod="openshift-service-ca/service-ca-79f68b9f4f-r6xjw"
Jul 13 18:51:02.759617 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:02.759582    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-57cd4\" (UniqueName: \"kubernetes.io/projected/f49e0954-6f1a-4f69-888c-662c5dd14c14-kube-api-access-57cd4\") pod \"service-ca-79f68b9f4f-r6xjw\" (UID: \"f49e0954-6f1a-4f69-888c-662c5dd14c14\") " pod="openshift-service-ca/service-ca-79f68b9f4f-r6xjw"
Jul 13 18:51:02.846457 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:02.846358    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1aa6cfb3-d5a7-4504-a9d6-11d63dd133f7-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-rjnwt\" (UID: \"1aa6cfb3-d5a7-4504-a9d6-11d63dd133f7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-rjnwt"
Jul 13 18:51:02.846457 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:02.846435    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ae45caa3-cc5b-4100-809e-2961c7fab65c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-whnln\" (UID: \"ae45caa3-cc5b-4100-809e-2961c7fab65c\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-whnln"
Jul 13 18:51:02.846692 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:51:02.846548    2565 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 13 18:51:02.846692 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:51:02.846651    2565 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 13 18:51:02.846836 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:51:02.846715    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1aa6cfb3-d5a7-4504-a9d6-11d63dd133f7-samples-operator-tls podName:1aa6cfb3-d5a7-4504-a9d6-11d63dd133f7 nodeName:}" failed. No retries permitted until 2026-07-13 18:51:10.846680348 +0000 UTC m=+57.371813747 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/1aa6cfb3-d5a7-4504-a9d6-11d63dd133f7-samples-operator-tls") pod "cluster-samples-operator-65db858894-rjnwt" (UID: "1aa6cfb3-d5a7-4504-a9d6-11d63dd133f7") : secret "samples-operator-tls" not found
Jul 13 18:51:02.846836 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:51:02.846745    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ae45caa3-cc5b-4100-809e-2961c7fab65c-cluster-monitoring-operator-tls podName:ae45caa3-cc5b-4100-809e-2961c7fab65c nodeName:}" failed. No retries permitted until 2026-07-13 18:51:10.846726978 +0000 UTC m=+57.371860376 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/ae45caa3-cc5b-4100-809e-2961c7fab65c-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-whnln" (UID: "ae45caa3-cc5b-4100-809e-2961c7fab65c") : secret "cluster-monitoring-operator-tls" not found
Jul 13 18:51:02.913931 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:02.913882    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-79f68b9f4f-r6xjw"
Jul 13 18:51:03.106897 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:03.106814    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-79f68b9f4f-r6xjw"]
Jul 13 18:51:03.109791 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:51:03.109749    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf49e0954_6f1a_4f69_888c_662c5dd14c14.slice/crio-62297f37bdc063706287f34332e79dedab6819934d1194832428178cc7e339a6 WatchSource:0}: Error finding container 62297f37bdc063706287f34332e79dedab6819934d1194832428178cc7e339a6: Status 404 returned error can't find the container with id 62297f37bdc063706287f34332e79dedab6819934d1194832428178cc7e339a6
Jul 13 18:51:03.269663 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:03.269625    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-4rt5g_2ddf2ee1-c9d7-4f30-aa52-b94c6d250c70/serve-healthcheck-canary/0.log"
Jul 13 18:51:03.301591 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:03.301543    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-79f68b9f4f-r6xjw" event={"ID":"f49e0954-6f1a-4f69-888c-662c5dd14c14","Type":"ContainerStarted","Data":"15d3baeaf31218328de2068e8733bf19815518623e96bc1c6a7eca0fc701949e"}
Jul 13 18:51:03.302091 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:03.301644    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-79f68b9f4f-r6xjw" event={"ID":"f49e0954-6f1a-4f69-888c-662c5dd14c14","Type":"ContainerStarted","Data":"62297f37bdc063706287f34332e79dedab6819934d1194832428178cc7e339a6"}
Jul 13 18:51:03.321268 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:03.321210    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-79f68b9f4f-r6xjw" podStartSLOduration=1.321188788 podStartE2EDuration="1.321188788s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 18:51:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:51:03.32067499 +0000 UTC m=+49.845808405" watchObservedRunningTime="2026-07-13 18:51:03.321188788 +0000 UTC m=+49.846322207"
Jul 13 18:51:03.451802 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:03.451762    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/053446da-3c51-46b2-a759-aad264c7b0f9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xmzjq\" (UID: \"053446da-3c51-46b2-a759-aad264c7b0f9\") " pod="openshift-insights/insights-runtime-extractor-xmzjq"
Jul 13 18:51:03.451984 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:03.451822    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/053446da-3c51-46b2-a759-aad264c7b0f9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xmzjq\" (UID: \"053446da-3c51-46b2-a759-aad264c7b0f9\") " pod="openshift-insights/insights-runtime-extractor-xmzjq"
Jul 13 18:51:03.451984 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:51:03.451920    2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 13 18:51:03.451984 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:51:03.451974    2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 13 18:51:03.452132 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:51:03.452003    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/053446da-3c51-46b2-a759-aad264c7b0f9-kube-rbac-proxy-cm podName:053446da-3c51-46b2-a759-aad264c7b0f9 nodeName:}" failed. No retries permitted until 2026-07-13 18:51:07.451979845 +0000 UTC m=+53.977113242 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/053446da-3c51-46b2-a759-aad264c7b0f9-kube-rbac-proxy-cm") pod "insights-runtime-extractor-xmzjq" (UID: "053446da-3c51-46b2-a759-aad264c7b0f9") : configmap "kube-rbac-proxy" not found
Jul 13 18:51:03.452132 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:51:03.452023    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/053446da-3c51-46b2-a759-aad264c7b0f9-insights-runtime-extractor-tls podName:053446da-3c51-46b2-a759-aad264c7b0f9 nodeName:}" failed. No retries permitted until 2026-07-13 18:51:07.452013266 +0000 UTC m=+53.977146663 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/053446da-3c51-46b2-a759-aad264c7b0f9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xmzjq" (UID: "053446da-3c51-46b2-a759-aad264c7b0f9") : secret "insights-runtime-extractor-tls" not found
Jul 13 18:51:03.957905 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:03.957857    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a10721d0-fc1e-485e-848b-7b877ad97903-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-bdj79\" (UID: \"a10721d0-fc1e-485e-848b-7b877ad97903\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-bdj79"
Jul 13 18:51:03.958090 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:51:03.958032    2565 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 13 18:51:03.958134 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:51:03.958117    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a10721d0-fc1e-485e-848b-7b877ad97903-networking-console-plugin-cert podName:a10721d0-fc1e-485e-848b-7b877ad97903 nodeName:}" failed. No retries permitted until 2026-07-13 18:51:11.958095056 +0000 UTC m=+58.483228454 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/a10721d0-fc1e-485e-848b-7b877ad97903-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-bdj79" (UID: "a10721d0-fc1e-485e-848b-7b877ad97903") : secret "networking-console-plugin-cert" not found
Jul 13 18:51:04.306693 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:04.306657    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-bjfpf" event={"ID":"f4e3f129-6df0-4398-ad91-634ac530dba1","Type":"ContainerStarted","Data":"67855f4f7261eb0251875c65259773d9d723430d3d32c09247f90f2945c94d76"}
Jul 13 18:51:04.307206 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:04.306730    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-bjfpf" event={"ID":"f4e3f129-6df0-4398-ad91-634ac530dba1","Type":"ContainerStarted","Data":"ac694735a4046339360b6233b0147da5c08bc3b3c846ff8c806a2530bb00f470"}
Jul 13 18:51:04.326351 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:04.326297    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-bjfpf" podStartSLOduration=1.570432606 podStartE2EDuration="3.32627959s" totalImagesPullingTime="1.755846984s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 18:51:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:51:04.325945904 +0000 UTC m=+50.851079320" watchObservedRunningTime="2026-07-13 18:51:04.32627959 +0000 UTC m=+50.851413007"
Jul 13 18:51:07.488903 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:07.488859    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/053446da-3c51-46b2-a759-aad264c7b0f9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xmzjq\" (UID: \"053446da-3c51-46b2-a759-aad264c7b0f9\") " pod="openshift-insights/insights-runtime-extractor-xmzjq"
Jul 13 18:51:07.488903 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:07.488910    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/053446da-3c51-46b2-a759-aad264c7b0f9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xmzjq\" (UID: \"053446da-3c51-46b2-a759-aad264c7b0f9\") " pod="openshift-insights/insights-runtime-extractor-xmzjq"
Jul 13 18:51:07.489422 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:51:07.489000    2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 13 18:51:07.489422 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:51:07.489023    2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 13 18:51:07.489422 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:51:07.489071    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/053446da-3c51-46b2-a759-aad264c7b0f9-kube-rbac-proxy-cm podName:053446da-3c51-46b2-a759-aad264c7b0f9 nodeName:}" failed. No retries permitted until 2026-07-13 18:51:15.489054406 +0000 UTC m=+62.014187799 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/053446da-3c51-46b2-a759-aad264c7b0f9-kube-rbac-proxy-cm") pod "insights-runtime-extractor-xmzjq" (UID: "053446da-3c51-46b2-a759-aad264c7b0f9") : configmap "kube-rbac-proxy" not found
Jul 13 18:51:07.489422 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:51:07.489086    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/053446da-3c51-46b2-a759-aad264c7b0f9-insights-runtime-extractor-tls podName:053446da-3c51-46b2-a759-aad264c7b0f9 nodeName:}" failed. No retries permitted until 2026-07-13 18:51:15.489079896 +0000 UTC m=+62.014213290 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/053446da-3c51-46b2-a759-aad264c7b0f9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xmzjq" (UID: "053446da-3c51-46b2-a759-aad264c7b0f9") : secret "insights-runtime-extractor-tls" not found
Jul 13 18:51:07.690774 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:07.690724    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5a144761-c470-4ac9-95cb-a629c9aac64a-metrics-tls\") pod \"dns-default-jw7lc\" (UID: \"5a144761-c470-4ac9-95cb-a629c9aac64a\") " pod="openshift-dns/dns-default-jw7lc"
Jul 13 18:51:07.693299 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:07.693273    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5a144761-c470-4ac9-95cb-a629c9aac64a-metrics-tls\") pod \"dns-default-jw7lc\" (UID: \"5a144761-c470-4ac9-95cb-a629c9aac64a\") " pod="openshift-dns/dns-default-jw7lc"
Jul 13 18:51:07.772878 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:07.772837    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-jw7lc"
Jul 13 18:51:07.942486 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:07.942445    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-jw7lc"]
Jul 13 18:51:07.945170 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:51:07.945136    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a144761_c470_4ac9_95cb_a629c9aac64a.slice/crio-be49ada9b044e992d3f6774730a02366c40142eb4ead4c20e59852a72601d3cd WatchSource:0}: Error finding container be49ada9b044e992d3f6774730a02366c40142eb4ead4c20e59852a72601d3cd: Status 404 returned error can't find the container with id be49ada9b044e992d3f6774730a02366c40142eb4ead4c20e59852a72601d3cd
Jul 13 18:51:08.323322 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:08.323285    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-jw7lc" event={"ID":"5a144761-c470-4ac9-95cb-a629c9aac64a","Type":"ContainerStarted","Data":"be49ada9b044e992d3f6774730a02366c40142eb4ead4c20e59852a72601d3cd"}
Jul 13 18:51:10.331514 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:10.331473    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-jw7lc" event={"ID":"5a144761-c470-4ac9-95cb-a629c9aac64a","Type":"ContainerStarted","Data":"443c4ad4dfa10fc82cb5c8ab5efc366b7684af20b4d017c75436fa59cfa9658f"}
Jul 13 18:51:10.331852 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:10.331538    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-jw7lc" event={"ID":"5a144761-c470-4ac9-95cb-a629c9aac64a","Type":"ContainerStarted","Data":"5c0ea33c3c6ade133b06b436de6b9c8c3fc9a7c75f1098f6341f06e891e7d4f3"}
Jul 13 18:51:10.917925 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:10.917887    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1aa6cfb3-d5a7-4504-a9d6-11d63dd133f7-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-rjnwt\" (UID: \"1aa6cfb3-d5a7-4504-a9d6-11d63dd133f7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-rjnwt"
Jul 13 18:51:10.918102 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:10.917938    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ae45caa3-cc5b-4100-809e-2961c7fab65c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-whnln\" (UID: \"ae45caa3-cc5b-4100-809e-2961c7fab65c\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-whnln"
Jul 13 18:51:10.918102 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:51:10.918033    2565 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 13 18:51:10.918102 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:51:10.918092    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ae45caa3-cc5b-4100-809e-2961c7fab65c-cluster-monitoring-operator-tls podName:ae45caa3-cc5b-4100-809e-2961c7fab65c nodeName:}" failed. No retries permitted until 2026-07-13 18:51:26.918076988 +0000 UTC m=+73.443210382 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/ae45caa3-cc5b-4100-809e-2961c7fab65c-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-whnln" (UID: "ae45caa3-cc5b-4100-809e-2961c7fab65c") : secret "cluster-monitoring-operator-tls" not found
Jul 13 18:51:10.920595 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:10.920568    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1aa6cfb3-d5a7-4504-a9d6-11d63dd133f7-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-rjnwt\" (UID: \"1aa6cfb3-d5a7-4504-a9d6-11d63dd133f7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-rjnwt"
Jul 13 18:51:10.926388 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:10.926350    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-rjnwt"
Jul 13 18:51:11.089836 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:11.089793    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-65db858894-rjnwt"]
Jul 13 18:51:11.335548 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:11.335509    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-rjnwt" event={"ID":"1aa6cfb3-d5a7-4504-a9d6-11d63dd133f7","Type":"ContainerStarted","Data":"d579ecd8b06081e87a7c5837d343dcc985a417c9651878248ef6a45f6892981e"}
Jul 13 18:51:11.336037 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:11.335637    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-jw7lc"
Jul 13 18:51:11.355925 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:11.355873    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-jw7lc" podStartSLOduration=18.260403349 podStartE2EDuration="20.355855186s" totalImagesPullingTime="2.095451837s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 18:50:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:51:11.354982636 +0000 UTC m=+57.880116053" watchObservedRunningTime="2026-07-13 18:51:11.355855186 +0000 UTC m=+57.880988602"
Jul 13 18:51:12.029540 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:12.029489    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a10721d0-fc1e-485e-848b-7b877ad97903-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-bdj79\" (UID: \"a10721d0-fc1e-485e-848b-7b877ad97903\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-bdj79"
Jul 13 18:51:12.032525 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:12.032491    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a10721d0-fc1e-485e-848b-7b877ad97903-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-bdj79\" (UID: \"a10721d0-fc1e-485e-848b-7b877ad97903\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-bdj79"
Jul 13 18:51:12.213122 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:12.213089    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6cf856758c-bdj79"
Jul 13 18:51:12.362596 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:12.362547    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6cf856758c-bdj79"]
Jul 13 18:51:12.365881 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:51:12.365845    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda10721d0_fc1e_485e_848b_7b877ad97903.slice/crio-b6027fed8498a746813d307d42204606fd084ad7d7409e792de7adbee49e4ec4 WatchSource:0}: Error finding container b6027fed8498a746813d307d42204606fd084ad7d7409e792de7adbee49e4ec4: Status 404 returned error can't find the container with id b6027fed8498a746813d307d42204606fd084ad7d7409e792de7adbee49e4ec4
Jul 13 18:51:13.347584 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:13.347545    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-rjnwt" event={"ID":"1aa6cfb3-d5a7-4504-a9d6-11d63dd133f7","Type":"ContainerStarted","Data":"a6fea1eb05803491638729d2001f4956ba8a6b156109726f9a401611e9f388b4"}
Jul 13 18:51:13.347811 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:13.347610    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-rjnwt" event={"ID":"1aa6cfb3-d5a7-4504-a9d6-11d63dd133f7","Type":"ContainerStarted","Data":"a70cb6ca301bd8d71c829a4127cd6719bd3520d2324dc295e7d8c7e2155de078"}
Jul 13 18:51:13.349302 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:13.349271    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6cf856758c-bdj79" event={"ID":"a10721d0-fc1e-485e-848b-7b877ad97903","Type":"ContainerStarted","Data":"b6027fed8498a746813d307d42204606fd084ad7d7409e792de7adbee49e4ec4"}
Jul 13 18:51:13.372258 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:13.372202    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-rjnwt" podStartSLOduration=17.676597125 podStartE2EDuration="19.372184519s" totalImagesPullingTime="1.695587394s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 18:50:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:51:13.371402825 +0000 UTC m=+59.896536262" watchObservedRunningTime="2026-07-13 18:51:13.372184519 +0000 UTC m=+59.897317934"
Jul 13 18:51:14.354165 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:14.354121    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6cf856758c-bdj79" event={"ID":"a10721d0-fc1e-485e-848b-7b877ad97903","Type":"ContainerStarted","Data":"2f333cc80df257f2227719d111c810a28965e86f0db91aaa5ca15c741de13e08"}
Jul 13 18:51:14.371079 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:14.370978    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-6cf856758c-bdj79" podStartSLOduration=18.176850008 podStartE2EDuration="19.370957871s" totalImagesPullingTime="1.194107863s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 18:50:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:51:14.370276201 +0000 UTC m=+60.895409637" watchObservedRunningTime="2026-07-13 18:51:14.370957871 +0000 UTC m=+60.896091288"
Jul 13 18:51:15.564186 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:15.564147    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/053446da-3c51-46b2-a759-aad264c7b0f9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xmzjq\" (UID: \"053446da-3c51-46b2-a759-aad264c7b0f9\") " pod="openshift-insights/insights-runtime-extractor-xmzjq"
Jul 13 18:51:15.564186 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:15.564190    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/053446da-3c51-46b2-a759-aad264c7b0f9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xmzjq\" (UID: \"053446da-3c51-46b2-a759-aad264c7b0f9\") " pod="openshift-insights/insights-runtime-extractor-xmzjq"
Jul 13 18:51:15.564728 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:15.564679    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/053446da-3c51-46b2-a759-aad264c7b0f9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xmzjq\" (UID: \"053446da-3c51-46b2-a759-aad264c7b0f9\") " pod="openshift-insights/insights-runtime-extractor-xmzjq"
Jul 13 18:51:15.566776 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:15.566753    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/053446da-3c51-46b2-a759-aad264c7b0f9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xmzjq\" (UID: \"053446da-3c51-46b2-a759-aad264c7b0f9\") " pod="openshift-insights/insights-runtime-extractor-xmzjq"
Jul 13 18:51:15.673192 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:15.673157    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-c5j2r\""
Jul 13 18:51:15.680712 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:15.680665    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-xmzjq"
Jul 13 18:51:15.827615 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:15.827527    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-xmzjq"]
Jul 13 18:51:15.830813 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:51:15.830784    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod053446da_3c51_46b2_a759_aad264c7b0f9.slice/crio-6859a65d666ab0d273deb461d8b29406134565f87d99a5f7272a831932c5efef WatchSource:0}: Error finding container 6859a65d666ab0d273deb461d8b29406134565f87d99a5f7272a831932c5efef: Status 404 returned error can't find the container with id 6859a65d666ab0d273deb461d8b29406134565f87d99a5f7272a831932c5efef
Jul 13 18:51:16.365251 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:16.365209    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xmzjq" event={"ID":"053446da-3c51-46b2-a759-aad264c7b0f9","Type":"ContainerStarted","Data":"6859a65d666ab0d273deb461d8b29406134565f87d99a5f7272a831932c5efef"}
Jul 13 18:51:17.243813 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:17.243786    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2bscc"
Jul 13 18:51:19.699729 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:19.699662    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d4fd4b59-2ecd-497b-a485-1e94101a9033-metrics-certs\") pod \"network-metrics-daemon-4sl67\" (UID: \"d4fd4b59-2ecd-497b-a485-1e94101a9033\") " pod="openshift-multus/network-metrics-daemon-4sl67"
Jul 13 18:51:19.702626 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:19.702606    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 13 18:51:19.713371 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:19.713336    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d4fd4b59-2ecd-497b-a485-1e94101a9033-metrics-certs\") pod \"network-metrics-daemon-4sl67\" (UID: \"d4fd4b59-2ecd-497b-a485-1e94101a9033\") " pod="openshift-multus/network-metrics-daemon-4sl67"
Jul 13 18:51:19.771904 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:19.771870    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-gsq9l\""
Jul 13 18:51:19.779417 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:19.779381    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4sl67"
Jul 13 18:51:19.800896 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:19.800867    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6bbc0748-e539-4679-a100-1ba45794129b-original-pull-secret\") pod \"global-pull-secret-syncer-qlzr5\" (UID: \"6bbc0748-e539-4679-a100-1ba45794129b\") " pod="kube-system/global-pull-secret-syncer-qlzr5"
Jul 13 18:51:19.803972 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:19.803953    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 13 18:51:19.813747 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:19.813693    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6bbc0748-e539-4679-a100-1ba45794129b-original-pull-secret\") pod \"global-pull-secret-syncer-qlzr5\" (UID: \"6bbc0748-e539-4679-a100-1ba45794129b\") " pod="kube-system/global-pull-secret-syncer-qlzr5"
Jul 13 18:51:19.902124 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:19.902090    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvvhf\" (UniqueName: \"kubernetes.io/projected/76869c1f-1537-4e3e-b7e2-654a08cd2376-kube-api-access-qvvhf\") pod \"network-check-target-5tgm8\" (UID: \"76869c1f-1537-4e3e-b7e2-654a08cd2376\") " pod="openshift-network-diagnostics/network-check-target-5tgm8"
Jul 13 18:51:19.904745 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:19.904687    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvvhf\" (UniqueName: \"kubernetes.io/projected/76869c1f-1537-4e3e-b7e2-654a08cd2376-kube-api-access-qvvhf\") pod \"network-check-target-5tgm8\" (UID: \"76869c1f-1537-4e3e-b7e2-654a08cd2376\") " pod="openshift-network-diagnostics/network-check-target-5tgm8"
Jul 13 18:51:19.927872 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:19.927846    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-4sl67"]
Jul 13 18:51:19.929892 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:51:19.929864    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4fd4b59_2ecd_497b_a485_1e94101a9033.slice/crio-906e2fa1bc6f4418eddd06de7445c22e4e63a1cb47fef72db21cc7401c295b52 WatchSource:0}: Error finding container 906e2fa1bc6f4418eddd06de7445c22e4e63a1cb47fef72db21cc7401c295b52: Status 404 returned error can't find the container with id 906e2fa1bc6f4418eddd06de7445c22e4e63a1cb47fef72db21cc7401c295b52
Jul 13 18:51:20.057729 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:20.057678    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-dz6tv\""
Jul 13 18:51:20.062096 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:20.062074    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qlzr5"
Jul 13 18:51:20.064636 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:20.064598    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5tgm8"
Jul 13 18:51:20.241426 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:20.241289    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-qlzr5"]
Jul 13 18:51:20.265365 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:20.265317    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-5tgm8"]
Jul 13 18:51:20.287467 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:51:20.287434    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bbc0748_e539_4679_a100_1ba45794129b.slice/crio-111d6bb0dc88be9e4a25f98244ca8e502b7e25fb059f206dc166b17922998677 WatchSource:0}: Error finding container 111d6bb0dc88be9e4a25f98244ca8e502b7e25fb059f206dc166b17922998677: Status 404 returned error can't find the container with id 111d6bb0dc88be9e4a25f98244ca8e502b7e25fb059f206dc166b17922998677
Jul 13 18:51:20.288166 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:51:20.288143    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76869c1f_1537_4e3e_b7e2_654a08cd2376.slice/crio-99c77a049ce0eaf92e0d16f64d9d8ce1dbf8a70544cbd890028cf233b4718803 WatchSource:0}: Error finding container 99c77a049ce0eaf92e0d16f64d9d8ce1dbf8a70544cbd890028cf233b4718803: Status 404 returned error can't find the container with id 99c77a049ce0eaf92e0d16f64d9d8ce1dbf8a70544cbd890028cf233b4718803
Jul 13 18:51:20.385502 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:20.385404    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-4sl67" event={"ID":"d4fd4b59-2ecd-497b-a485-1e94101a9033","Type":"ContainerStarted","Data":"906e2fa1bc6f4418eddd06de7445c22e4e63a1cb47fef72db21cc7401c295b52"}
Jul 13 18:51:20.387086 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:20.387052    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-5tgm8" event={"ID":"76869c1f-1537-4e3e-b7e2-654a08cd2376","Type":"ContainerStarted","Data":"afe6bc07ac31c2ce89ce7ef395ac37c3a9060d028fffb8117cbc04d9e0119d79"}
Jul 13 18:51:20.387241 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:20.387107    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-5tgm8" event={"ID":"76869c1f-1537-4e3e-b7e2-654a08cd2376","Type":"ContainerStarted","Data":"99c77a049ce0eaf92e0d16f64d9d8ce1dbf8a70544cbd890028cf233b4718803"}
Jul 13 18:51:20.387241 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:20.387135    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-5tgm8"
Jul 13 18:51:20.388924 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:20.388894    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xmzjq" event={"ID":"053446da-3c51-46b2-a759-aad264c7b0f9","Type":"ContainerStarted","Data":"78658166eced79a67c062b1a783aee4fad4f93a3b76a843601ef66a5dfea5830"}
Jul 13 18:51:20.389074 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:20.388954    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xmzjq" event={"ID":"053446da-3c51-46b2-a759-aad264c7b0f9","Type":"ContainerStarted","Data":"50eac669b2ed997786bd7f2e5546adf348f81f504107ce394ec9a545d62d6a8d"}
Jul 13 18:51:20.390106 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:20.390077    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-qlzr5" event={"ID":"6bbc0748-e539-4679-a100-1ba45794129b","Type":"ContainerStarted","Data":"111d6bb0dc88be9e4a25f98244ca8e502b7e25fb059f206dc166b17922998677"}
Jul 13 18:51:20.409874 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:20.409807    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-5tgm8" podStartSLOduration=65.409782059 podStartE2EDuration="1m5.409782059s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 18:50:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:51:20.409018271 +0000 UTC m=+66.934151717" watchObservedRunningTime="2026-07-13 18:51:20.409782059 +0000 UTC m=+66.934915474"
Jul 13 18:51:21.344546 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:21.344508    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-jw7lc"
Jul 13 18:51:21.397791 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:21.397748    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-4sl67" event={"ID":"d4fd4b59-2ecd-497b-a485-1e94101a9033","Type":"ContainerStarted","Data":"ec8418dc0a029680aded481c6c8c33676cc1c4a7df4915838296d7f2fd17bbca"}
Jul 13 18:51:21.398015 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:21.397831    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-4sl67" event={"ID":"d4fd4b59-2ecd-497b-a485-1e94101a9033","Type":"ContainerStarted","Data":"b106030dba195d42eaad394708fddbab4e506cb359fa6e15685f2fae9b53bbc5"}
Jul 13 18:51:21.418012 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:21.417936    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-4sl67" podStartSLOduration=66.488888948 podStartE2EDuration="1m7.41790949s" totalImagesPullingTime="929.020542ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 18:50:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:51:21.416579823 +0000 UTC m=+67.941713258" watchObservedRunningTime="2026-07-13 18:51:21.41790949 +0000 UTC m=+67.943042907"
Jul 13 18:51:25.421685 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:25.421575    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xmzjq" event={"ID":"053446da-3c51-46b2-a759-aad264c7b0f9","Type":"ContainerStarted","Data":"0cc6face1bd436105405b7604caba024dcee0fb17779813ac131bfe65cd1062f"}
Jul 13 18:51:25.423083 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:25.423057    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-qlzr5" event={"ID":"6bbc0748-e539-4679-a100-1ba45794129b","Type":"ContainerStarted","Data":"e8c508e8aadb7f766e41976a2a941c142f2e228f7636d6cfd9b13c0c6e9736b3"}
Jul 13 18:51:25.450875 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:25.450827    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-xmzjq" podStartSLOduration=17.301101876 podStartE2EDuration="26.450809101s" totalImagesPullingTime="9.149707225s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 18:50:59 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:51:25.449692679 +0000 UTC m=+71.974826111" watchObservedRunningTime="2026-07-13 18:51:25.450809101 +0000 UTC m=+71.975942517"
Jul 13 18:51:26.961788 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:26.961746    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ae45caa3-cc5b-4100-809e-2961c7fab65c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-whnln\" (UID: \"ae45caa3-cc5b-4100-809e-2961c7fab65c\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-whnln"
Jul 13 18:51:26.964273 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:26.964248    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ae45caa3-cc5b-4100-809e-2961c7fab65c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-whnln\" (UID: \"ae45caa3-cc5b-4100-809e-2961c7fab65c\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-whnln"
Jul 13 18:51:27.126397 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:27.126366    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-zxfcj\""
Jul 13 18:51:27.131803 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:27.131775    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-whnln"
Jul 13 18:51:27.325775 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:27.325697    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-qlzr5" podStartSLOduration=67.511256071 podStartE2EDuration="1m12.325677955s" totalImagesPullingTime="4.814421884s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 18:50:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:51:25.479460413 +0000 UTC m=+72.004593831" watchObservedRunningTime="2026-07-13 18:51:27.325677955 +0000 UTC m=+73.850811370"
Jul 13 18:51:27.326364 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:27.326318    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-whnln"]
Jul 13 18:51:27.330195 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:51:27.330164    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae45caa3_cc5b_4100_809e_2961c7fab65c.slice/crio-77b6820beaf12472de2c66def57e92dac03182a13c17a45e36a591ff657c9873 WatchSource:0}: Error finding container 77b6820beaf12472de2c66def57e92dac03182a13c17a45e36a591ff657c9873: Status 404 returned error can't find the container with id 77b6820beaf12472de2c66def57e92dac03182a13c17a45e36a591ff657c9873
Jul 13 18:51:27.430760 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:27.430689    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-whnln" event={"ID":"ae45caa3-cc5b-4100-809e-2961c7fab65c","Type":"ContainerStarted","Data":"77b6820beaf12472de2c66def57e92dac03182a13c17a45e36a591ff657c9873"}
Jul 13 18:51:28.065046 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:28.065003    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7d57676984-24sqj"]
Jul 13 18:51:28.098044 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:28.098008    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7d57676984-24sqj"]
Jul 13 18:51:28.098215 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:28.098113    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7d57676984-24sqj"
Jul 13 18:51:28.101738 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:28.101690    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-p56b4\""
Jul 13 18:51:28.101993 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:28.101975    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 13 18:51:28.102082 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:28.102015    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 13 18:51:28.102842 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:28.102823    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 13 18:51:28.107527 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:28.107503    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 13 18:51:28.169282 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:28.169242    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8a5ea70c-0be6-434d-81a5-7c72ac3bd61f-bound-sa-token\") pod \"image-registry-7d57676984-24sqj\" (UID: \"8a5ea70c-0be6-434d-81a5-7c72ac3bd61f\") " pod="openshift-image-registry/image-registry-7d57676984-24sqj"
Jul 13 18:51:28.169282 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:28.169284    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kv9qk\" (UniqueName: \"kubernetes.io/projected/8a5ea70c-0be6-434d-81a5-7c72ac3bd61f-kube-api-access-kv9qk\") pod \"image-registry-7d57676984-24sqj\" (UID: \"8a5ea70c-0be6-434d-81a5-7c72ac3bd61f\") " pod="openshift-image-registry/image-registry-7d57676984-24sqj"
Jul 13 18:51:28.169546 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:28.169309    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8a5ea70c-0be6-434d-81a5-7c72ac3bd61f-trusted-ca\") pod \"image-registry-7d57676984-24sqj\" (UID: \"8a5ea70c-0be6-434d-81a5-7c72ac3bd61f\") " pod="openshift-image-registry/image-registry-7d57676984-24sqj"
Jul 13 18:51:28.169546 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:28.169417    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8a5ea70c-0be6-434d-81a5-7c72ac3bd61f-image-registry-private-configuration\") pod \"image-registry-7d57676984-24sqj\" (UID: \"8a5ea70c-0be6-434d-81a5-7c72ac3bd61f\") " pod="openshift-image-registry/image-registry-7d57676984-24sqj"
Jul 13 18:51:28.169546 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:28.169468    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8a5ea70c-0be6-434d-81a5-7c72ac3bd61f-ca-trust-extracted\") pod \"image-registry-7d57676984-24sqj\" (UID: \"8a5ea70c-0be6-434d-81a5-7c72ac3bd61f\") " pod="openshift-image-registry/image-registry-7d57676984-24sqj"
Jul 13 18:51:28.169546 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:28.169493    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8a5ea70c-0be6-434d-81a5-7c72ac3bd61f-installation-pull-secrets\") pod \"image-registry-7d57676984-24sqj\" (UID: \"8a5ea70c-0be6-434d-81a5-7c72ac3bd61f\") " pod="openshift-image-registry/image-registry-7d57676984-24sqj"
Jul 13 18:51:28.169546 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:28.169535    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8a5ea70c-0be6-434d-81a5-7c72ac3bd61f-registry-tls\") pod \"image-registry-7d57676984-24sqj\" (UID: \"8a5ea70c-0be6-434d-81a5-7c72ac3bd61f\") " pod="openshift-image-registry/image-registry-7d57676984-24sqj"
Jul 13 18:51:28.169766 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:28.169573    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8a5ea70c-0be6-434d-81a5-7c72ac3bd61f-registry-certificates\") pod \"image-registry-7d57676984-24sqj\" (UID: \"8a5ea70c-0be6-434d-81a5-7c72ac3bd61f\") " pod="openshift-image-registry/image-registry-7d57676984-24sqj"
Jul 13 18:51:28.270363 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:28.270322    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8a5ea70c-0be6-434d-81a5-7c72ac3bd61f-image-registry-private-configuration\") pod \"image-registry-7d57676984-24sqj\" (UID: \"8a5ea70c-0be6-434d-81a5-7c72ac3bd61f\") " pod="openshift-image-registry/image-registry-7d57676984-24sqj"
Jul 13 18:51:28.270552 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:28.270370    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8a5ea70c-0be6-434d-81a5-7c72ac3bd61f-ca-trust-extracted\") pod \"image-registry-7d57676984-24sqj\" (UID: \"8a5ea70c-0be6-434d-81a5-7c72ac3bd61f\") " pod="openshift-image-registry/image-registry-7d57676984-24sqj"
Jul 13 18:51:28.270552 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:28.270399    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8a5ea70c-0be6-434d-81a5-7c72ac3bd61f-installation-pull-secrets\") pod \"image-registry-7d57676984-24sqj\" (UID: \"8a5ea70c-0be6-434d-81a5-7c72ac3bd61f\") " pod="openshift-image-registry/image-registry-7d57676984-24sqj"
Jul 13 18:51:28.270552 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:28.270436    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8a5ea70c-0be6-434d-81a5-7c72ac3bd61f-registry-tls\") pod \"image-registry-7d57676984-24sqj\" (UID: \"8a5ea70c-0be6-434d-81a5-7c72ac3bd61f\") " pod="openshift-image-registry/image-registry-7d57676984-24sqj"
Jul 13 18:51:28.270552 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:28.270478    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8a5ea70c-0be6-434d-81a5-7c72ac3bd61f-registry-certificates\") pod \"image-registry-7d57676984-24sqj\" (UID: \"8a5ea70c-0be6-434d-81a5-7c72ac3bd61f\") " pod="openshift-image-registry/image-registry-7d57676984-24sqj"
Jul 13 18:51:28.270810 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:28.270630    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8a5ea70c-0be6-434d-81a5-7c72ac3bd61f-bound-sa-token\") pod \"image-registry-7d57676984-24sqj\" (UID: \"8a5ea70c-0be6-434d-81a5-7c72ac3bd61f\") " pod="openshift-image-registry/image-registry-7d57676984-24sqj"
Jul 13 18:51:28.270810 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:28.270687    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kv9qk\" (UniqueName: \"kubernetes.io/projected/8a5ea70c-0be6-434d-81a5-7c72ac3bd61f-kube-api-access-kv9qk\") pod \"image-registry-7d57676984-24sqj\" (UID: \"8a5ea70c-0be6-434d-81a5-7c72ac3bd61f\") " pod="openshift-image-registry/image-registry-7d57676984-24sqj"
Jul 13 18:51:28.270810 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:28.270761    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8a5ea70c-0be6-434d-81a5-7c72ac3bd61f-trusted-ca\") pod \"image-registry-7d57676984-24sqj\" (UID: \"8a5ea70c-0be6-434d-81a5-7c72ac3bd61f\") " pod="openshift-image-registry/image-registry-7d57676984-24sqj"
Jul 13 18:51:28.270976 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:28.270839    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8a5ea70c-0be6-434d-81a5-7c72ac3bd61f-ca-trust-extracted\") pod \"image-registry-7d57676984-24sqj\" (UID: \"8a5ea70c-0be6-434d-81a5-7c72ac3bd61f\") " pod="openshift-image-registry/image-registry-7d57676984-24sqj"
Jul 13 18:51:28.271401 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:28.271378    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8a5ea70c-0be6-434d-81a5-7c72ac3bd61f-registry-certificates\") pod \"image-registry-7d57676984-24sqj\" (UID: \"8a5ea70c-0be6-434d-81a5-7c72ac3bd61f\") " pod="openshift-image-registry/image-registry-7d57676984-24sqj"
Jul 13 18:51:28.271910 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:28.271884    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8a5ea70c-0be6-434d-81a5-7c72ac3bd61f-trusted-ca\") pod \"image-registry-7d57676984-24sqj\" (UID: \"8a5ea70c-0be6-434d-81a5-7c72ac3bd61f\") " pod="openshift-image-registry/image-registry-7d57676984-24sqj"
Jul 13 18:51:28.273281 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:28.273254    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8a5ea70c-0be6-434d-81a5-7c72ac3bd61f-registry-tls\") pod \"image-registry-7d57676984-24sqj\" (UID: \"8a5ea70c-0be6-434d-81a5-7c72ac3bd61f\") " pod="openshift-image-registry/image-registry-7d57676984-24sqj"
Jul 13 18:51:28.273367 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:28.273262    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8a5ea70c-0be6-434d-81a5-7c72ac3bd61f-installation-pull-secrets\") pod \"image-registry-7d57676984-24sqj\" (UID: \"8a5ea70c-0be6-434d-81a5-7c72ac3bd61f\") " pod="openshift-image-registry/image-registry-7d57676984-24sqj"
Jul 13 18:51:28.273367 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:28.273330    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8a5ea70c-0be6-434d-81a5-7c72ac3bd61f-image-registry-private-configuration\") pod \"image-registry-7d57676984-24sqj\" (UID: \"8a5ea70c-0be6-434d-81a5-7c72ac3bd61f\") " pod="openshift-image-registry/image-registry-7d57676984-24sqj"
Jul 13 18:51:28.302872 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:28.302831    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8a5ea70c-0be6-434d-81a5-7c72ac3bd61f-bound-sa-token\") pod \"image-registry-7d57676984-24sqj\" (UID: \"8a5ea70c-0be6-434d-81a5-7c72ac3bd61f\") " pod="openshift-image-registry/image-registry-7d57676984-24sqj"
Jul 13 18:51:28.304393 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:28.304357    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kv9qk\" (UniqueName: \"kubernetes.io/projected/8a5ea70c-0be6-434d-81a5-7c72ac3bd61f-kube-api-access-kv9qk\") pod \"image-registry-7d57676984-24sqj\" (UID: \"8a5ea70c-0be6-434d-81a5-7c72ac3bd61f\") " pod="openshift-image-registry/image-registry-7d57676984-24sqj"
Jul 13 18:51:28.408577 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:28.408473    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7d57676984-24sqj"
Jul 13 18:51:28.589122 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:28.589080    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7d57676984-24sqj"]
Jul 13 18:51:28.594171 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:51:28.594140    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a5ea70c_0be6_434d_81a5_7c72ac3bd61f.slice/crio-c00932d8da17eb3323c2ab1c2062f22ade2bd0f1d250a5ce47ab3320365d8e14 WatchSource:0}: Error finding container c00932d8da17eb3323c2ab1c2062f22ade2bd0f1d250a5ce47ab3320365d8e14: Status 404 returned error can't find the container with id c00932d8da17eb3323c2ab1c2062f22ade2bd0f1d250a5ce47ab3320365d8e14
Jul 13 18:51:29.442552 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:29.442507    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7d57676984-24sqj" event={"ID":"8a5ea70c-0be6-434d-81a5-7c72ac3bd61f","Type":"ContainerStarted","Data":"7b75ea1f618359aa5e2e3e593912555f18283098c6ce63c640a7a983c4f97455"}
Jul 13 18:51:29.443022 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:29.442573    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7d57676984-24sqj" event={"ID":"8a5ea70c-0be6-434d-81a5-7c72ac3bd61f","Type":"ContainerStarted","Data":"c00932d8da17eb3323c2ab1c2062f22ade2bd0f1d250a5ce47ab3320365d8e14"}
Jul 13 18:51:29.443022 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:29.442617    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7d57676984-24sqj"
Jul 13 18:51:29.471774 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:29.471689    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-7d57676984-24sqj" podStartSLOduration=1.471670076 podStartE2EDuration="1.471670076s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 18:51:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:51:29.469502362 +0000 UTC m=+75.994635806" watchObservedRunningTime="2026-07-13 18:51:29.471670076 +0000 UTC m=+75.996803491"
Jul 13 18:51:30.448362 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:30.448324    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-whnln" event={"ID":"ae45caa3-cc5b-4100-809e-2961c7fab65c","Type":"ContainerStarted","Data":"beca69568eb181fdad6e8510bdd0af82e7a89c72f25969bf2c1e519bcb6b7dac"}
Jul 13 18:51:30.478508 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:30.478438    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-whnln" podStartSLOduration=34.32030388 podStartE2EDuration="36.478421167s" totalImagesPullingTime="2.158117287s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 18:50:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:51:30.477998499 +0000 UTC m=+77.003131916" watchObservedRunningTime="2026-07-13 18:51:30.478421167 +0000 UTC m=+77.003554583"
Jul 13 18:51:37.460992 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:37.460861    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-64785b6675-dbsvm"]
Jul 13 18:51:37.464181 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:37.464159    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-64785b6675-dbsvm"
Jul 13 18:51:37.467683 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:37.467655    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Jul 13 18:51:37.468712 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:37.468684    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 13 18:51:37.468793 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:37.468743    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-t7kcs\""
Jul 13 18:51:37.468844 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:37.468748    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Jul 13 18:51:37.478566 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:37.478537    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-64785b6675-dbsvm"]
Jul 13 18:51:37.539161 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:37.539117    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/5e935794-9a93-4a39-b78d-1e973574ad31-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-dbsvm\" (UID: \"5e935794-9a93-4a39-b78d-1e973574ad31\") " pod="openshift-monitoring/prometheus-operator-64785b6675-dbsvm"
Jul 13 18:51:37.539161 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:37.539165    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5e935794-9a93-4a39-b78d-1e973574ad31-metrics-client-ca\") pod \"prometheus-operator-64785b6675-dbsvm\" (UID: \"5e935794-9a93-4a39-b78d-1e973574ad31\") " pod="openshift-monitoring/prometheus-operator-64785b6675-dbsvm"
Jul 13 18:51:37.539423 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:37.539225    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrqdj\" (UniqueName: \"kubernetes.io/projected/5e935794-9a93-4a39-b78d-1e973574ad31-kube-api-access-lrqdj\") pod \"prometheus-operator-64785b6675-dbsvm\" (UID: \"5e935794-9a93-4a39-b78d-1e973574ad31\") " pod="openshift-monitoring/prometheus-operator-64785b6675-dbsvm"
Jul 13 18:51:37.539423 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:37.539279    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5e935794-9a93-4a39-b78d-1e973574ad31-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-64785b6675-dbsvm\" (UID: \"5e935794-9a93-4a39-b78d-1e973574ad31\") " pod="openshift-monitoring/prometheus-operator-64785b6675-dbsvm"
Jul 13 18:51:37.640565 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:37.640526    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/5e935794-9a93-4a39-b78d-1e973574ad31-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-dbsvm\" (UID: \"5e935794-9a93-4a39-b78d-1e973574ad31\") " pod="openshift-monitoring/prometheus-operator-64785b6675-dbsvm"
Jul 13 18:51:37.640565 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:37.640577    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5e935794-9a93-4a39-b78d-1e973574ad31-metrics-client-ca\") pod \"prometheus-operator-64785b6675-dbsvm\" (UID: \"5e935794-9a93-4a39-b78d-1e973574ad31\") " pod="openshift-monitoring/prometheus-operator-64785b6675-dbsvm"
Jul 13 18:51:37.640884 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:37.640609    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrqdj\" (UniqueName: \"kubernetes.io/projected/5e935794-9a93-4a39-b78d-1e973574ad31-kube-api-access-lrqdj\") pod \"prometheus-operator-64785b6675-dbsvm\" (UID: \"5e935794-9a93-4a39-b78d-1e973574ad31\") " pod="openshift-monitoring/prometheus-operator-64785b6675-dbsvm"
Jul 13 18:51:37.640884 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:37.640645    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5e935794-9a93-4a39-b78d-1e973574ad31-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-64785b6675-dbsvm\" (UID: \"5e935794-9a93-4a39-b78d-1e973574ad31\") " pod="openshift-monitoring/prometheus-operator-64785b6675-dbsvm"
Jul 13 18:51:37.641269 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:37.641246    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5e935794-9a93-4a39-b78d-1e973574ad31-metrics-client-ca\") pod \"prometheus-operator-64785b6675-dbsvm\" (UID: \"5e935794-9a93-4a39-b78d-1e973574ad31\") " pod="openshift-monitoring/prometheus-operator-64785b6675-dbsvm"
Jul 13 18:51:37.643485 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:37.643457    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5e935794-9a93-4a39-b78d-1e973574ad31-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-64785b6675-dbsvm\" (UID: \"5e935794-9a93-4a39-b78d-1e973574ad31\") " pod="openshift-monitoring/prometheus-operator-64785b6675-dbsvm"
Jul 13 18:51:37.643604 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:37.643467    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/5e935794-9a93-4a39-b78d-1e973574ad31-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-dbsvm\" (UID: \"5e935794-9a93-4a39-b78d-1e973574ad31\") " pod="openshift-monitoring/prometheus-operator-64785b6675-dbsvm"
Jul 13 18:51:37.649402 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:37.649370    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrqdj\" (UniqueName: \"kubernetes.io/projected/5e935794-9a93-4a39-b78d-1e973574ad31-kube-api-access-lrqdj\") pod \"prometheus-operator-64785b6675-dbsvm\" (UID: \"5e935794-9a93-4a39-b78d-1e973574ad31\") " pod="openshift-monitoring/prometheus-operator-64785b6675-dbsvm"
Jul 13 18:51:37.775499 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:37.775468    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-64785b6675-dbsvm"
Jul 13 18:51:37.913231 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:37.913196    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-64785b6675-dbsvm"]
Jul 13 18:51:37.916364 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:51:37.916326    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e935794_9a93_4a39_b78d_1e973574ad31.slice/crio-5320fc25d5168ee2e5252589c8e660d7c0d99c7041ab4721a4caa3b183e4a6cb WatchSource:0}: Error finding container 5320fc25d5168ee2e5252589c8e660d7c0d99c7041ab4721a4caa3b183e4a6cb: Status 404 returned error can't find the container with id 5320fc25d5168ee2e5252589c8e660d7c0d99c7041ab4721a4caa3b183e4a6cb
Jul 13 18:51:38.477984 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:38.477951    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-64785b6675-dbsvm" event={"ID":"5e935794-9a93-4a39-b78d-1e973574ad31","Type":"ContainerStarted","Data":"5320fc25d5168ee2e5252589c8e660d7c0d99c7041ab4721a4caa3b183e4a6cb"}
Jul 13 18:51:39.483680 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:39.483644    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-64785b6675-dbsvm" event={"ID":"5e935794-9a93-4a39-b78d-1e973574ad31","Type":"ContainerStarted","Data":"abe6eca760200aeea1c0fe11ddab421b0343297af89e4be1dd09f99d50374320"}
Jul 13 18:51:40.489225 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:40.489190    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-64785b6675-dbsvm" event={"ID":"5e935794-9a93-4a39-b78d-1e973574ad31","Type":"ContainerStarted","Data":"5cbb488403b680d6344ae57fcadb9a5218c1ea4da36d4ce21708dd228f3350be"}
Jul 13 18:51:42.967211 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:42.967140    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-64785b6675-dbsvm" podStartSLOduration=4.514837842 podStartE2EDuration="5.967121971s" totalImagesPullingTime="1.452284129s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 18:51:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:51:40.518895459 +0000 UTC m=+87.044028874" watchObservedRunningTime="2026-07-13 18:51:42.967121971 +0000 UTC m=+89.492255388"
Jul 13 18:51:42.967776 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:42.967375    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-s5sd7"]
Jul 13 18:51:42.972966 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:42.972935    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-s5sd7"
Jul 13 18:51:42.976644 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:42.976614    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 13 18:51:42.976886 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:42.976628    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 13 18:51:42.977089 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:42.976669    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 13 18:51:42.977228 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:42.977209    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-zmqhc\""
Jul 13 18:51:43.087215 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:43.087170    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d984d1bc-572a-4897-a32b-af3f919735f2-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-s5sd7\" (UID: \"d984d1bc-572a-4897-a32b-af3f919735f2\") " pod="openshift-monitoring/node-exporter-s5sd7"
Jul 13 18:51:43.087215 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:43.087219    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d984d1bc-572a-4897-a32b-af3f919735f2-node-exporter-tls\") pod \"node-exporter-s5sd7\" (UID: \"d984d1bc-572a-4897-a32b-af3f919735f2\") " pod="openshift-monitoring/node-exporter-s5sd7"
Jul 13 18:51:43.087468 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:43.087243    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d984d1bc-572a-4897-a32b-af3f919735f2-node-exporter-textfile\") pod \"node-exporter-s5sd7\" (UID: \"d984d1bc-572a-4897-a32b-af3f919735f2\") " pod="openshift-monitoring/node-exporter-s5sd7"
Jul 13 18:51:43.087468 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:43.087315    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d984d1bc-572a-4897-a32b-af3f919735f2-node-exporter-wtmp\") pod \"node-exporter-s5sd7\" (UID: \"d984d1bc-572a-4897-a32b-af3f919735f2\") " pod="openshift-monitoring/node-exporter-s5sd7"
Jul 13 18:51:43.087468 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:43.087379    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d984d1bc-572a-4897-a32b-af3f919735f2-sys\") pod \"node-exporter-s5sd7\" (UID: \"d984d1bc-572a-4897-a32b-af3f919735f2\") " pod="openshift-monitoring/node-exporter-s5sd7"
Jul 13 18:51:43.087468 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:43.087408    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d984d1bc-572a-4897-a32b-af3f919735f2-node-exporter-accelerators-collector-config\") pod \"node-exporter-s5sd7\" (UID: \"d984d1bc-572a-4897-a32b-af3f919735f2\") " pod="openshift-monitoring/node-exporter-s5sd7"
Jul 13 18:51:43.087468 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:43.087445    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d984d1bc-572a-4897-a32b-af3f919735f2-metrics-client-ca\") pod \"node-exporter-s5sd7\" (UID: \"d984d1bc-572a-4897-a32b-af3f919735f2\") " pod="openshift-monitoring/node-exporter-s5sd7"
Jul 13 18:51:43.087732 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:43.087469    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5r9m\" (UniqueName: \"kubernetes.io/projected/d984d1bc-572a-4897-a32b-af3f919735f2-kube-api-access-f5r9m\") pod \"node-exporter-s5sd7\" (UID: \"d984d1bc-572a-4897-a32b-af3f919735f2\") " pod="openshift-monitoring/node-exporter-s5sd7"
Jul 13 18:51:43.087732 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:43.087527    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d984d1bc-572a-4897-a32b-af3f919735f2-root\") pod \"node-exporter-s5sd7\" (UID: \"d984d1bc-572a-4897-a32b-af3f919735f2\") " pod="openshift-monitoring/node-exporter-s5sd7"
Jul 13 18:51:43.188976 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:43.188939    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d984d1bc-572a-4897-a32b-af3f919735f2-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-s5sd7\" (UID: \"d984d1bc-572a-4897-a32b-af3f919735f2\") " pod="openshift-monitoring/node-exporter-s5sd7"
Jul 13 18:51:43.188976 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:43.188979    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d984d1bc-572a-4897-a32b-af3f919735f2-node-exporter-tls\") pod \"node-exporter-s5sd7\" (UID: \"d984d1bc-572a-4897-a32b-af3f919735f2\") " pod="openshift-monitoring/node-exporter-s5sd7"
Jul 13 18:51:43.189224 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:43.189002    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d984d1bc-572a-4897-a32b-af3f919735f2-node-exporter-textfile\") pod \"node-exporter-s5sd7\" (UID: \"d984d1bc-572a-4897-a32b-af3f919735f2\") " pod="openshift-monitoring/node-exporter-s5sd7"
Jul 13 18:51:43.189224 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:43.189021    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d984d1bc-572a-4897-a32b-af3f919735f2-node-exporter-wtmp\") pod \"node-exporter-s5sd7\" (UID: \"d984d1bc-572a-4897-a32b-af3f919735f2\") " pod="openshift-monitoring/node-exporter-s5sd7"
Jul 13 18:51:43.189224 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:43.189049    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d984d1bc-572a-4897-a32b-af3f919735f2-sys\") pod \"node-exporter-s5sd7\" (UID: \"d984d1bc-572a-4897-a32b-af3f919735f2\") " pod="openshift-monitoring/node-exporter-s5sd7"
Jul 13 18:51:43.189224 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:43.189097    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d984d1bc-572a-4897-a32b-af3f919735f2-node-exporter-accelerators-collector-config\") pod \"node-exporter-s5sd7\" (UID: \"d984d1bc-572a-4897-a32b-af3f919735f2\") " pod="openshift-monitoring/node-exporter-s5sd7"
Jul 13 18:51:43.189224 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:51:43.189132    2565 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 13 18:51:43.189224 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:43.189184    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d984d1bc-572a-4897-a32b-af3f919735f2-sys\") pod \"node-exporter-s5sd7\" (UID: \"d984d1bc-572a-4897-a32b-af3f919735f2\") " pod="openshift-monitoring/node-exporter-s5sd7"
Jul 13 18:51:43.189224 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:43.189194    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d984d1bc-572a-4897-a32b-af3f919735f2-node-exporter-wtmp\") pod \"node-exporter-s5sd7\" (UID: \"d984d1bc-572a-4897-a32b-af3f919735f2\") " pod="openshift-monitoring/node-exporter-s5sd7"
Jul 13 18:51:43.189224 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:51:43.189209    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d984d1bc-572a-4897-a32b-af3f919735f2-node-exporter-tls podName:d984d1bc-572a-4897-a32b-af3f919735f2 nodeName:}" failed. No retries permitted until 2026-07-13 18:51:43.689186852 +0000 UTC m=+90.214320252 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/d984d1bc-572a-4897-a32b-af3f919735f2-node-exporter-tls") pod "node-exporter-s5sd7" (UID: "d984d1bc-572a-4897-a32b-af3f919735f2") : secret "node-exporter-tls" not found
Jul 13 18:51:43.189633 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:43.189250    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d984d1bc-572a-4897-a32b-af3f919735f2-metrics-client-ca\") pod \"node-exporter-s5sd7\" (UID: \"d984d1bc-572a-4897-a32b-af3f919735f2\") " pod="openshift-monitoring/node-exporter-s5sd7"
Jul 13 18:51:43.189633 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:43.189393    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d984d1bc-572a-4897-a32b-af3f919735f2-node-exporter-textfile\") pod \"node-exporter-s5sd7\" (UID: \"d984d1bc-572a-4897-a32b-af3f919735f2\") " pod="openshift-monitoring/node-exporter-s5sd7"
Jul 13 18:51:43.189633 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:43.189476    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5r9m\" (UniqueName: \"kubernetes.io/projected/d984d1bc-572a-4897-a32b-af3f919735f2-kube-api-access-f5r9m\") pod \"node-exporter-s5sd7\" (UID: \"d984d1bc-572a-4897-a32b-af3f919735f2\") " pod="openshift-monitoring/node-exporter-s5sd7"
Jul 13 18:51:43.189633 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:43.189512    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d984d1bc-572a-4897-a32b-af3f919735f2-root\") pod \"node-exporter-s5sd7\" (UID: \"d984d1bc-572a-4897-a32b-af3f919735f2\") " pod="openshift-monitoring/node-exporter-s5sd7"
Jul 13 18:51:43.189633 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:43.189604    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d984d1bc-572a-4897-a32b-af3f919735f2-root\") pod \"node-exporter-s5sd7\" (UID: \"d984d1bc-572a-4897-a32b-af3f919735f2\") " pod="openshift-monitoring/node-exporter-s5sd7"
Jul 13 18:51:43.189915 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:43.189847    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d984d1bc-572a-4897-a32b-af3f919735f2-node-exporter-accelerators-collector-config\") pod \"node-exporter-s5sd7\" (UID: \"d984d1bc-572a-4897-a32b-af3f919735f2\") " pod="openshift-monitoring/node-exporter-s5sd7"
Jul 13 18:51:43.189915 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:43.189872    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d984d1bc-572a-4897-a32b-af3f919735f2-metrics-client-ca\") pod \"node-exporter-s5sd7\" (UID: \"d984d1bc-572a-4897-a32b-af3f919735f2\") " pod="openshift-monitoring/node-exporter-s5sd7"
Jul 13 18:51:43.192006 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:43.191985    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d984d1bc-572a-4897-a32b-af3f919735f2-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-s5sd7\" (UID: \"d984d1bc-572a-4897-a32b-af3f919735f2\") " pod="openshift-monitoring/node-exporter-s5sd7"
Jul 13 18:51:43.213716 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:43.213666    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5r9m\" (UniqueName: \"kubernetes.io/projected/d984d1bc-572a-4897-a32b-af3f919735f2-kube-api-access-f5r9m\") pod \"node-exporter-s5sd7\" (UID: \"d984d1bc-572a-4897-a32b-af3f919735f2\") " pod="openshift-monitoring/node-exporter-s5sd7"
Jul 13 18:51:43.693247 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:43.693205    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d984d1bc-572a-4897-a32b-af3f919735f2-node-exporter-tls\") pod \"node-exporter-s5sd7\" (UID: \"d984d1bc-572a-4897-a32b-af3f919735f2\") " pod="openshift-monitoring/node-exporter-s5sd7"
Jul 13 18:51:43.695754 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:43.695720    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d984d1bc-572a-4897-a32b-af3f919735f2-node-exporter-tls\") pod \"node-exporter-s5sd7\" (UID: \"d984d1bc-572a-4897-a32b-af3f919735f2\") " pod="openshift-monitoring/node-exporter-s5sd7"
Jul 13 18:51:43.885630 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:43.885597    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-s5sd7"
Jul 13 18:51:43.900774 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:51:43.900686    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd984d1bc_572a_4897_a32b_af3f919735f2.slice/crio-14ba89f36e0cb7f1d6d25c36f6954875f44606204169fb26f48f01ca1e5c912c WatchSource:0}: Error finding container 14ba89f36e0cb7f1d6d25c36f6954875f44606204169fb26f48f01ca1e5c912c: Status 404 returned error can't find the container with id 14ba89f36e0cb7f1d6d25c36f6954875f44606204169fb26f48f01ca1e5c912c
Jul 13 18:51:43.932475 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:43.932440    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 13 18:51:43.937645 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:43.937616    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 13 18:51:43.941794 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:43.941757    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Jul 13 18:51:43.941965 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:43.941920    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Jul 13 18:51:43.942122 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:43.942084    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Jul 13 18:51:43.942257 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:43.942177    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Jul 13 18:51:43.942257 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:43.942216    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-m2cff\""
Jul 13 18:51:43.942257 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:43.942240    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Jul 13 18:51:43.942403 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:43.942337    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Jul 13 18:51:43.942403 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:43.942398    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Jul 13 18:51:43.942503 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:43.942470    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Jul 13 18:51:43.943777 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:43.943686    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Jul 13 18:51:43.960198 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:43.960168    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 13 18:51:44.096341 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:44.096293    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/882acd32-9687-4c39-9d84-4893191bccd3-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"882acd32-9687-4c39-9d84-4893191bccd3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 13 18:51:44.096341 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:44.096344    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/882acd32-9687-4c39-9d84-4893191bccd3-tls-assets\") pod \"alertmanager-main-0\" (UID: \"882acd32-9687-4c39-9d84-4893191bccd3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 13 18:51:44.096896 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:44.096362    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/882acd32-9687-4c39-9d84-4893191bccd3-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"882acd32-9687-4c39-9d84-4893191bccd3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 13 18:51:44.096896 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:44.096393    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/882acd32-9687-4c39-9d84-4893191bccd3-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"882acd32-9687-4c39-9d84-4893191bccd3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 13 18:51:44.096896 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:44.096412    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tds24\" (UniqueName: \"kubernetes.io/projected/882acd32-9687-4c39-9d84-4893191bccd3-kube-api-access-tds24\") pod \"alertmanager-main-0\" (UID: \"882acd32-9687-4c39-9d84-4893191bccd3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 13 18:51:44.096896 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:44.096475    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/882acd32-9687-4c39-9d84-4893191bccd3-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"882acd32-9687-4c39-9d84-4893191bccd3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 13 18:51:44.096896 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:44.096501    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/882acd32-9687-4c39-9d84-4893191bccd3-config-volume\") pod \"alertmanager-main-0\" (UID: \"882acd32-9687-4c39-9d84-4893191bccd3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 13 18:51:44.096896 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:44.096526    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/882acd32-9687-4c39-9d84-4893191bccd3-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"882acd32-9687-4c39-9d84-4893191bccd3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 13 18:51:44.096896 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:44.096545    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/882acd32-9687-4c39-9d84-4893191bccd3-web-config\") pod \"alertmanager-main-0\" (UID: \"882acd32-9687-4c39-9d84-4893191bccd3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 13 18:51:44.096896 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:44.096560    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/882acd32-9687-4c39-9d84-4893191bccd3-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"882acd32-9687-4c39-9d84-4893191bccd3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 13 18:51:44.096896 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:44.096599    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/882acd32-9687-4c39-9d84-4893191bccd3-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"882acd32-9687-4c39-9d84-4893191bccd3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 13 18:51:44.096896 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:44.096638    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/882acd32-9687-4c39-9d84-4893191bccd3-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"882acd32-9687-4c39-9d84-4893191bccd3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 13 18:51:44.096896 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:44.096667    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/882acd32-9687-4c39-9d84-4893191bccd3-config-out\") pod \"alertmanager-main-0\" (UID: \"882acd32-9687-4c39-9d84-4893191bccd3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 13 18:51:44.198202 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:44.198093    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/882acd32-9687-4c39-9d84-4893191bccd3-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"882acd32-9687-4c39-9d84-4893191bccd3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 13 18:51:44.198202 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:44.198147    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/882acd32-9687-4c39-9d84-4893191bccd3-tls-assets\") pod \"alertmanager-main-0\" (UID: \"882acd32-9687-4c39-9d84-4893191bccd3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 13 18:51:44.198202 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:44.198175    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/882acd32-9687-4c39-9d84-4893191bccd3-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"882acd32-9687-4c39-9d84-4893191bccd3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 13 18:51:44.198488 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:44.198211    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/882acd32-9687-4c39-9d84-4893191bccd3-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"882acd32-9687-4c39-9d84-4893191bccd3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 13 18:51:44.198488 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:44.198239    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tds24\" (UniqueName: \"kubernetes.io/projected/882acd32-9687-4c39-9d84-4893191bccd3-kube-api-access-tds24\") pod \"alertmanager-main-0\" (UID: \"882acd32-9687-4c39-9d84-4893191bccd3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 13 18:51:44.198488 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:44.198283    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/882acd32-9687-4c39-9d84-4893191bccd3-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"882acd32-9687-4c39-9d84-4893191bccd3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 13 18:51:44.198488 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:44.198313    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/882acd32-9687-4c39-9d84-4893191bccd3-config-volume\") pod \"alertmanager-main-0\" (UID: \"882acd32-9687-4c39-9d84-4893191bccd3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 13 18:51:44.198488 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:44.198336    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/882acd32-9687-4c39-9d84-4893191bccd3-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"882acd32-9687-4c39-9d84-4893191bccd3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 13 18:51:44.198488 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:44.198359    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/882acd32-9687-4c39-9d84-4893191bccd3-web-config\") pod \"alertmanager-main-0\" (UID: \"882acd32-9687-4c39-9d84-4893191bccd3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 13 18:51:44.198488 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:44.198381    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/882acd32-9687-4c39-9d84-4893191bccd3-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"882acd32-9687-4c39-9d84-4893191bccd3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 13 18:51:44.198488 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:44.198406    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/882acd32-9687-4c39-9d84-4893191bccd3-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"882acd32-9687-4c39-9d84-4893191bccd3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 13 18:51:44.198488 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:44.198442    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/882acd32-9687-4c39-9d84-4893191bccd3-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"882acd32-9687-4c39-9d84-4893191bccd3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 13 18:51:44.198488 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:44.198483    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/882acd32-9687-4c39-9d84-4893191bccd3-config-out\") pod \"alertmanager-main-0\" (UID: \"882acd32-9687-4c39-9d84-4893191bccd3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 13 18:51:44.199534 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:44.199501    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/882acd32-9687-4c39-9d84-4893191bccd3-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"882acd32-9687-4c39-9d84-4893191bccd3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 13 18:51:44.199632 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:44.199501    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/882acd32-9687-4c39-9d84-4893191bccd3-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"882acd32-9687-4c39-9d84-4893191bccd3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 13 18:51:44.199992 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:44.199964    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/882acd32-9687-4c39-9d84-4893191bccd3-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"882acd32-9687-4c39-9d84-4893191bccd3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 13 18:51:44.202599 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:44.202573    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/882acd32-9687-4c39-9d84-4893191bccd3-config-out\") pod \"alertmanager-main-0\" (UID: \"882acd32-9687-4c39-9d84-4893191bccd3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 13 18:51:44.202796 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:44.202773    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/882acd32-9687-4c39-9d84-4893191bccd3-web-config\") pod \"alertmanager-main-0\" (UID: \"882acd32-9687-4c39-9d84-4893191bccd3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 13 18:51:44.203061 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:44.203029    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/882acd32-9687-4c39-9d84-4893191bccd3-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"882acd32-9687-4c39-9d84-4893191bccd3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 13 18:51:44.203334 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:44.203293    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/882acd32-9687-4c39-9d84-4893191bccd3-config-volume\") pod \"alertmanager-main-0\" (UID: \"882acd32-9687-4c39-9d84-4893191bccd3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 13 18:51:44.203334 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:44.203323    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/882acd32-9687-4c39-9d84-4893191bccd3-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"882acd32-9687-4c39-9d84-4893191bccd3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 13 18:51:44.203961 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:44.203933    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/882acd32-9687-4c39-9d84-4893191bccd3-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"882acd32-9687-4c39-9d84-4893191bccd3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 13 18:51:44.204102 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:44.204071    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/882acd32-9687-4c39-9d84-4893191bccd3-tls-assets\") pod \"alertmanager-main-0\" (UID: \"882acd32-9687-4c39-9d84-4893191bccd3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 13 18:51:44.205149 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:44.205065    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/882acd32-9687-4c39-9d84-4893191bccd3-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"882acd32-9687-4c39-9d84-4893191bccd3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 13 18:51:44.205674 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:44.205649    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/882acd32-9687-4c39-9d84-4893191bccd3-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"882acd32-9687-4c39-9d84-4893191bccd3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 13 18:51:44.209676 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:44.209649    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tds24\" (UniqueName: \"kubernetes.io/projected/882acd32-9687-4c39-9d84-4893191bccd3-kube-api-access-tds24\") pod \"alertmanager-main-0\" (UID: \"882acd32-9687-4c39-9d84-4893191bccd3\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 13 18:51:44.249145 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:44.249109    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 13 18:51:44.415947 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:44.415860    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 13 18:51:44.419597 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:51:44.419563    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod882acd32_9687_4c39_9d84_4893191bccd3.slice/crio-44d22af040c71313966e6c708827472cb2ab28f5cb7c38a3f3d6c1d31f38f0fe WatchSource:0}: Error finding container 44d22af040c71313966e6c708827472cb2ab28f5cb7c38a3f3d6c1d31f38f0fe: Status 404 returned error can't find the container with id 44d22af040c71313966e6c708827472cb2ab28f5cb7c38a3f3d6c1d31f38f0fe
Jul 13 18:51:44.509992 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:44.509895    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"882acd32-9687-4c39-9d84-4893191bccd3","Type":"ContainerStarted","Data":"44d22af040c71313966e6c708827472cb2ab28f5cb7c38a3f3d6c1d31f38f0fe"}
Jul 13 18:51:44.511261 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:44.511239    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-s5sd7" event={"ID":"d984d1bc-572a-4897-a32b-af3f919735f2","Type":"ContainerStarted","Data":"14ba89f36e0cb7f1d6d25c36f6954875f44606204169fb26f48f01ca1e5c912c"}
Jul 13 18:51:44.923668 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:44.923622    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-5d9d8df5d-6dwcd"]
Jul 13 18:51:44.928972 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:44.928944    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-5d9d8df5d-6dwcd"
Jul 13 18:51:44.932059 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:44.932027    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Jul 13 18:51:44.932679 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:44.932662    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Jul 13 18:51:44.933357 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:44.933338    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Jul 13 18:51:44.933357 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:44.933349    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Jul 13 18:51:44.933478 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:44.933350    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Jul 13 18:51:44.933732 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:44.933695    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-lhpcf\""
Jul 13 18:51:44.934099 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:44.934079    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-8jd791hrqgqsj\""
Jul 13 18:51:44.951057 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:44.951027    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-5d9d8df5d-6dwcd"]
Jul 13 18:51:45.007513 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:45.007473    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/52addd15-9664-48bb-81a2-a2c683c03624-secret-thanos-querier-tls\") pod \"thanos-querier-5d9d8df5d-6dwcd\" (UID: \"52addd15-9664-48bb-81a2-a2c683c03624\") " pod="openshift-monitoring/thanos-querier-5d9d8df5d-6dwcd"
Jul 13 18:51:45.007733 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:45.007526    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/52addd15-9664-48bb-81a2-a2c683c03624-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5d9d8df5d-6dwcd\" (UID: \"52addd15-9664-48bb-81a2-a2c683c03624\") " pod="openshift-monitoring/thanos-querier-5d9d8df5d-6dwcd"
Jul 13 18:51:45.007733 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:45.007566    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtq57\" (UniqueName: \"kubernetes.io/projected/52addd15-9664-48bb-81a2-a2c683c03624-kube-api-access-vtq57\") pod \"thanos-querier-5d9d8df5d-6dwcd\" (UID: \"52addd15-9664-48bb-81a2-a2c683c03624\") " pod="openshift-monitoring/thanos-querier-5d9d8df5d-6dwcd"
Jul 13 18:51:45.007733 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:45.007594    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/52addd15-9664-48bb-81a2-a2c683c03624-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5d9d8df5d-6dwcd\" (UID: \"52addd15-9664-48bb-81a2-a2c683c03624\") " pod="openshift-monitoring/thanos-querier-5d9d8df5d-6dwcd"
Jul 13 18:51:45.007733 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:45.007680    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/52addd15-9664-48bb-81a2-a2c683c03624-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5d9d8df5d-6dwcd\" (UID: \"52addd15-9664-48bb-81a2-a2c683c03624\") " pod="openshift-monitoring/thanos-querier-5d9d8df5d-6dwcd"
Jul 13 18:51:45.007906 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:45.007744    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/52addd15-9664-48bb-81a2-a2c683c03624-metrics-client-ca\") pod \"thanos-querier-5d9d8df5d-6dwcd\" (UID: \"52addd15-9664-48bb-81a2-a2c683c03624\") " pod="openshift-monitoring/thanos-querier-5d9d8df5d-6dwcd"
Jul 13 18:51:45.007906 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:45.007816    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/52addd15-9664-48bb-81a2-a2c683c03624-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5d9d8df5d-6dwcd\" (UID: \"52addd15-9664-48bb-81a2-a2c683c03624\") " pod="openshift-monitoring/thanos-querier-5d9d8df5d-6dwcd"
Jul 13 18:51:45.007906 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:45.007850    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/52addd15-9664-48bb-81a2-a2c683c03624-secret-grpc-tls\") pod \"thanos-querier-5d9d8df5d-6dwcd\" (UID: \"52addd15-9664-48bb-81a2-a2c683c03624\") " pod="openshift-monitoring/thanos-querier-5d9d8df5d-6dwcd"
Jul 13 18:51:45.108684 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:45.108646    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/52addd15-9664-48bb-81a2-a2c683c03624-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5d9d8df5d-6dwcd\" (UID: \"52addd15-9664-48bb-81a2-a2c683c03624\") " pod="openshift-monitoring/thanos-querier-5d9d8df5d-6dwcd"
Jul 13 18:51:45.108684 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:45.108696    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/52addd15-9664-48bb-81a2-a2c683c03624-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5d9d8df5d-6dwcd\" (UID: \"52addd15-9664-48bb-81a2-a2c683c03624\") " pod="openshift-monitoring/thanos-querier-5d9d8df5d-6dwcd"
Jul 13 18:51:45.109250 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:45.108743    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/52addd15-9664-48bb-81a2-a2c683c03624-metrics-client-ca\") pod \"thanos-querier-5d9d8df5d-6dwcd\" (UID: \"52addd15-9664-48bb-81a2-a2c683c03624\") " pod="openshift-monitoring/thanos-querier-5d9d8df5d-6dwcd"
Jul 13 18:51:45.109250 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:45.108782    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/52addd15-9664-48bb-81a2-a2c683c03624-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5d9d8df5d-6dwcd\" (UID: \"52addd15-9664-48bb-81a2-a2c683c03624\") " pod="openshift-monitoring/thanos-querier-5d9d8df5d-6dwcd"
Jul 13 18:51:45.109250 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:45.108811    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/52addd15-9664-48bb-81a2-a2c683c03624-secret-grpc-tls\") pod \"thanos-querier-5d9d8df5d-6dwcd\" (UID: \"52addd15-9664-48bb-81a2-a2c683c03624\") " pod="openshift-monitoring/thanos-querier-5d9d8df5d-6dwcd"
Jul 13 18:51:45.109250 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:45.108836    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/52addd15-9664-48bb-81a2-a2c683c03624-secret-thanos-querier-tls\") pod \"thanos-querier-5d9d8df5d-6dwcd\" (UID: \"52addd15-9664-48bb-81a2-a2c683c03624\") " pod="openshift-monitoring/thanos-querier-5d9d8df5d-6dwcd"
Jul 13 18:51:45.109250 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:45.108864    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/52addd15-9664-48bb-81a2-a2c683c03624-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5d9d8df5d-6dwcd\" (UID: \"52addd15-9664-48bb-81a2-a2c683c03624\") " pod="openshift-monitoring/thanos-querier-5d9d8df5d-6dwcd"
Jul 13 18:51:45.109250 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:45.108898    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtq57\" (UniqueName: \"kubernetes.io/projected/52addd15-9664-48bb-81a2-a2c683c03624-kube-api-access-vtq57\") pod \"thanos-querier-5d9d8df5d-6dwcd\" (UID: \"52addd15-9664-48bb-81a2-a2c683c03624\") " pod="openshift-monitoring/thanos-querier-5d9d8df5d-6dwcd"
Jul 13 18:51:45.109990 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:45.109900    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/52addd15-9664-48bb-81a2-a2c683c03624-metrics-client-ca\") pod \"thanos-querier-5d9d8df5d-6dwcd\" (UID: \"52addd15-9664-48bb-81a2-a2c683c03624\") " pod="openshift-monitoring/thanos-querier-5d9d8df5d-6dwcd"
Jul 13 18:51:45.112310 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:45.112271    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/52addd15-9664-48bb-81a2-a2c683c03624-secret-thanos-querier-tls\") pod \"thanos-querier-5d9d8df5d-6dwcd\" (UID: \"52addd15-9664-48bb-81a2-a2c683c03624\") " pod="openshift-monitoring/thanos-querier-5d9d8df5d-6dwcd"
Jul 13 18:51:45.112799 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:45.112771    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/52addd15-9664-48bb-81a2-a2c683c03624-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5d9d8df5d-6dwcd\" (UID: \"52addd15-9664-48bb-81a2-a2c683c03624\") " pod="openshift-monitoring/thanos-querier-5d9d8df5d-6dwcd"
Jul 13 18:51:45.112912 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:45.112872    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/52addd15-9664-48bb-81a2-a2c683c03624-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5d9d8df5d-6dwcd\" (UID: \"52addd15-9664-48bb-81a2-a2c683c03624\") " pod="openshift-monitoring/thanos-querier-5d9d8df5d-6dwcd"
Jul 13 18:51:45.112912 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:45.112891    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/52addd15-9664-48bb-81a2-a2c683c03624-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5d9d8df5d-6dwcd\" (UID: \"52addd15-9664-48bb-81a2-a2c683c03624\") " pod="openshift-monitoring/thanos-querier-5d9d8df5d-6dwcd"
Jul 13 18:51:45.113109 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:45.113086    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/52addd15-9664-48bb-81a2-a2c683c03624-secret-grpc-tls\") pod \"thanos-querier-5d9d8df5d-6dwcd\" (UID: \"52addd15-9664-48bb-81a2-a2c683c03624\") " pod="openshift-monitoring/thanos-querier-5d9d8df5d-6dwcd"
Jul 13 18:51:45.113760 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:45.113692    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/52addd15-9664-48bb-81a2-a2c683c03624-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5d9d8df5d-6dwcd\" (UID: \"52addd15-9664-48bb-81a2-a2c683c03624\") " pod="openshift-monitoring/thanos-querier-5d9d8df5d-6dwcd"
Jul 13 18:51:45.119360 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:45.119330    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtq57\" (UniqueName: \"kubernetes.io/projected/52addd15-9664-48bb-81a2-a2c683c03624-kube-api-access-vtq57\") pod \"thanos-querier-5d9d8df5d-6dwcd\" (UID: \"52addd15-9664-48bb-81a2-a2c683c03624\") " pod="openshift-monitoring/thanos-querier-5d9d8df5d-6dwcd"
Jul 13 18:51:45.240332 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:45.240008    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-5d9d8df5d-6dwcd"
Jul 13 18:51:45.422190 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:45.422144    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-5d9d8df5d-6dwcd"]
Jul 13 18:51:45.554722 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:51:45.554670    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52addd15_9664_48bb_81a2_a2c683c03624.slice/crio-738d670dd56a496bb7885bd10cf55ddf50e3949ce979813be3fa4257ee8ed8b2 WatchSource:0}: Error finding container 738d670dd56a496bb7885bd10cf55ddf50e3949ce979813be3fa4257ee8ed8b2: Status 404 returned error can't find the container with id 738d670dd56a496bb7885bd10cf55ddf50e3949ce979813be3fa4257ee8ed8b2
Jul 13 18:51:46.521670 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:46.521566    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5d9d8df5d-6dwcd" event={"ID":"52addd15-9664-48bb-81a2-a2c683c03624","Type":"ContainerStarted","Data":"738d670dd56a496bb7885bd10cf55ddf50e3949ce979813be3fa4257ee8ed8b2"}
Jul 13 18:51:46.523775 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:46.523740    2565 generic.go:356] "Generic (PLEG): container finished" podID="882acd32-9687-4c39-9d84-4893191bccd3" containerID="0447498ed817a0745ac7f850e776da80aa7c6f89bed915fc236d39ceca577ddf" exitCode=0
Jul 13 18:51:46.523916 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:46.523850    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"882acd32-9687-4c39-9d84-4893191bccd3","Type":"ContainerDied","Data":"0447498ed817a0745ac7f850e776da80aa7c6f89bed915fc236d39ceca577ddf"}
Jul 13 18:51:47.529743 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:47.529683    2565 generic.go:356] "Generic (PLEG): container finished" podID="d984d1bc-572a-4897-a32b-af3f919735f2" containerID="9c3a3c5c597179b31178dda75df671801ba112ea8ccbba0560a5562ecab83ee6" exitCode=0
Jul 13 18:51:47.529743 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:47.529734    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-s5sd7" event={"ID":"d984d1bc-572a-4897-a32b-af3f919735f2","Type":"ContainerDied","Data":"9c3a3c5c597179b31178dda75df671801ba112ea8ccbba0560a5562ecab83ee6"}
Jul 13 18:51:47.531879 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:47.531850    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5d9d8df5d-6dwcd" event={"ID":"52addd15-9664-48bb-81a2-a2c683c03624","Type":"ContainerStarted","Data":"b6b39fb64aaf865ac093a171d806d681e6082029385fdcaa6bdad200d02fc60d"}
Jul 13 18:51:47.639211 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:47.639181    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5f68df7f58-jmgtf"]
Jul 13 18:51:47.643085 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:47.643060    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-jmgtf"
Jul 13 18:51:47.649646 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:47.649618    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Jul 13 18:51:47.649788 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:47.649622    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-8xlwg\""
Jul 13 18:51:47.658973 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:47.658913    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5f68df7f58-jmgtf"]
Jul 13 18:51:47.732662 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:47.732525    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrvw2\" (UniqueName: \"kubernetes.io/projected/ab3e7251-ebd4-487c-bef3-ecdffa077837-kube-api-access-xrvw2\") pod \"monitoring-plugin-5f68df7f58-jmgtf\" (UID: \"ab3e7251-ebd4-487c-bef3-ecdffa077837\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-jmgtf"
Jul 13 18:51:47.732662 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:47.732604    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ab3e7251-ebd4-487c-bef3-ecdffa077837-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-jmgtf\" (UID: \"ab3e7251-ebd4-487c-bef3-ecdffa077837\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-jmgtf"
Jul 13 18:51:47.833274 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:47.833234    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ab3e7251-ebd4-487c-bef3-ecdffa077837-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-jmgtf\" (UID: \"ab3e7251-ebd4-487c-bef3-ecdffa077837\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-jmgtf"
Jul 13 18:51:47.833462 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:47.833312    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrvw2\" (UniqueName: \"kubernetes.io/projected/ab3e7251-ebd4-487c-bef3-ecdffa077837-kube-api-access-xrvw2\") pod \"monitoring-plugin-5f68df7f58-jmgtf\" (UID: \"ab3e7251-ebd4-487c-bef3-ecdffa077837\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-jmgtf"
Jul 13 18:51:47.835911 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:47.835889    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ab3e7251-ebd4-487c-bef3-ecdffa077837-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-jmgtf\" (UID: \"ab3e7251-ebd4-487c-bef3-ecdffa077837\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-jmgtf"
Jul 13 18:51:47.843766 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:47.843729    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrvw2\" (UniqueName: \"kubernetes.io/projected/ab3e7251-ebd4-487c-bef3-ecdffa077837-kube-api-access-xrvw2\") pod \"monitoring-plugin-5f68df7f58-jmgtf\" (UID: \"ab3e7251-ebd4-487c-bef3-ecdffa077837\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-jmgtf"
Jul 13 18:51:47.969232 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:47.969142    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-jmgtf"
Jul 13 18:51:48.418830 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:48.418538    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5f68df7f58-jmgtf"]
Jul 13 18:51:48.422119 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:51:48.422082    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab3e7251_ebd4_487c_bef3_ecdffa077837.slice/crio-5f0010668acc7258f1dfcddb6ccc5cf72cb4e03e0dec6b86fe3cba6da30dceeb WatchSource:0}: Error finding container 5f0010668acc7258f1dfcddb6ccc5cf72cb4e03e0dec6b86fe3cba6da30dceeb: Status 404 returned error can't find the container with id 5f0010668acc7258f1dfcddb6ccc5cf72cb4e03e0dec6b86fe3cba6da30dceeb
Jul 13 18:51:48.545555 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:48.545508    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5d9d8df5d-6dwcd" event={"ID":"52addd15-9664-48bb-81a2-a2c683c03624","Type":"ContainerStarted","Data":"51fe73a5032b4a53365aff6795664c0d01b1f3a9bab3f274210e19209c5182b1"}
Jul 13 18:51:48.546037 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:48.545617    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5d9d8df5d-6dwcd" event={"ID":"52addd15-9664-48bb-81a2-a2c683c03624","Type":"ContainerStarted","Data":"d9da03a845d7873425d20034aad6b548c9594d113156e5bf661aa2ca07702727"}
Jul 13 18:51:48.548248 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:48.548142    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"882acd32-9687-4c39-9d84-4893191bccd3","Type":"ContainerStarted","Data":"d690478c28a2f49e9eecd647956269f98a6b7e7b51316ddff6369cde46c66023"}
Jul 13 18:51:48.548248 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:48.548211    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"882acd32-9687-4c39-9d84-4893191bccd3","Type":"ContainerStarted","Data":"6bb3111530dc1bb69a0e79a93d26aeb26e6cc826d15707fa661fb541d2e0da61"}
Jul 13 18:51:48.549893 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:48.549844    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-jmgtf" event={"ID":"ab3e7251-ebd4-487c-bef3-ecdffa077837","Type":"ContainerStarted","Data":"5f0010668acc7258f1dfcddb6ccc5cf72cb4e03e0dec6b86fe3cba6da30dceeb"}
Jul 13 18:51:48.553046 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:48.553013    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-s5sd7" event={"ID":"d984d1bc-572a-4897-a32b-af3f919735f2","Type":"ContainerStarted","Data":"ed80362b23a4aebd3549752c341e0a88e8fc2c80a403b6068508c6df44af264f"}
Jul 13 18:51:48.553179 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:48.553074    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-s5sd7" event={"ID":"d984d1bc-572a-4897-a32b-af3f919735f2","Type":"ContainerStarted","Data":"3dedb9ff01893331d59ff45a59a08e994a20dddeeaf28468e4f36d2e2d35ed0d"}
Jul 13 18:51:48.588812 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:48.588737    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-s5sd7" podStartSLOduration=3.630287243 podStartE2EDuration="6.588693561s" totalImagesPullingTime="2.872331285s" totalInitContainerRuntime="86.075033ms" isStatefulPod=true podCreationTimestamp="2026-07-13 18:51:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:51:48.586671042 +0000 UTC m=+95.111804480" watchObservedRunningTime="2026-07-13 18:51:48.588693561 +0000 UTC m=+95.113826958"
Jul 13 18:51:49.565179 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:49.565130    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5d9d8df5d-6dwcd" event={"ID":"52addd15-9664-48bb-81a2-a2c683c03624","Type":"ContainerStarted","Data":"d66482d9752f9693b0fa735582c6ffb7c96c1a7cafe34b3d356399ee998379ea"}
Jul 13 18:51:49.565666 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:49.565208    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5d9d8df5d-6dwcd" event={"ID":"52addd15-9664-48bb-81a2-a2c683c03624","Type":"ContainerStarted","Data":"ab982cf2781a7196b519c1a259913cb417d9c9591e1a855156629313904931c7"}
Jul 13 18:51:49.565666 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:49.565241    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5d9d8df5d-6dwcd" event={"ID":"52addd15-9664-48bb-81a2-a2c683c03624","Type":"ContainerStarted","Data":"0ac7b8ff1329b03f1a49505fb71c5673e0836c8894b8179559a2d51806b3f9d8"}
Jul 13 18:51:49.565666 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:49.565410    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-5d9d8df5d-6dwcd"
Jul 13 18:51:49.569629 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:49.569592    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"882acd32-9687-4c39-9d84-4893191bccd3","Type":"ContainerStarted","Data":"a9b0808783015d5baf89d50c38891729d2a6c557adf6316cc933b02b458a2d17"}
Jul 13 18:51:49.569629 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:49.569670    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"882acd32-9687-4c39-9d84-4893191bccd3","Type":"ContainerStarted","Data":"1c3a82e894ac5dd2f369d0c873cab6ccddb20b91e97a12f953c5a880eddc1583"}
Jul 13 18:51:49.569629 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:49.569746    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"882acd32-9687-4c39-9d84-4893191bccd3","Type":"ContainerStarted","Data":"46728ec117baba0f3ce6fcb63da43374adfe6fb5c01227a7600b5d0dba127927"}
Jul 13 18:51:49.570030 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:49.569786    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"882acd32-9687-4c39-9d84-4893191bccd3","Type":"ContainerStarted","Data":"445b78855e74fcbcb395888dde2057cf1087d45d9bd8c3066d7af9250ce04570"}
Jul 13 18:51:49.608494 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:49.608441    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-5d9d8df5d-6dwcd" podStartSLOduration=2.713906761 podStartE2EDuration="5.608423592s" totalImagesPullingTime="2.894516831s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 18:51:44 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:51:49.607722428 +0000 UTC m=+96.132855870" watchObservedRunningTime="2026-07-13 18:51:49.608423592 +0000 UTC m=+96.133557008"
Jul 13 18:51:49.670012 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:49.669947    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.671487618 podStartE2EDuration="6.669926352s" totalImagesPullingTime="2.981605534s" totalInitContainerRuntime="16.8332ms" isStatefulPod=false podCreationTimestamp="2026-07-13 18:51:43 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:51:49.666946118 +0000 UTC m=+96.192079534" watchObservedRunningTime="2026-07-13 18:51:49.669926352 +0000 UTC m=+96.195059770"
Jul 13 18:51:50.452857 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:50.452829    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7d57676984-24sqj"
Jul 13 18:51:50.576191 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:50.576155    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-jmgtf" event={"ID":"ab3e7251-ebd4-487c-bef3-ecdffa077837","Type":"ContainerStarted","Data":"857be204fd094264e35f7e782fe916c674101aeb808bb0ed770041cb782df99e"}
Jul 13 18:51:50.576839 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:50.576818    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-jmgtf"
Jul 13 18:51:50.581827 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:50.581793    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-jmgtf"
Jul 13 18:51:50.598012 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:50.597953    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-jmgtf" podStartSLOduration=2.296090793 podStartE2EDuration="3.597933473s" totalImagesPullingTime="1.30184268s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 18:51:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:51:50.596946267 +0000 UTC m=+97.122079705" watchObservedRunningTime="2026-07-13 18:51:50.597933473 +0000 UTC m=+97.123066889"
Jul 13 18:51:51.401009 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:51.400973    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-5tgm8"
Jul 13 18:51:51.535857 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:51.535819    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-f45bd79cb-2gcbm"]
Jul 13 18:51:51.539295 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:51.539267    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-f45bd79cb-2gcbm"
Jul 13 18:51:51.543012 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:51.542987    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 13 18:51:51.543197 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:51.543172    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 13 18:51:51.543267 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:51.543171    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 13 18:51:51.543341 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:51.543284    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 13 18:51:51.543565 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:51.543547    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 13 18:51:51.543770 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:51.543753    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 13 18:51:51.543826 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:51.543802    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 13 18:51:51.544269 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:51.544246    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-gxdrl\""
Jul 13 18:51:51.549946 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:51.549921    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 13 18:51:51.551271 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:51.551248    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-f45bd79cb-2gcbm"]
Jul 13 18:51:51.677663 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:51.677576    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cd892f91-5591-49e8-bdfa-c455803db4fc-console-serving-cert\") pod \"console-f45bd79cb-2gcbm\" (UID: \"cd892f91-5591-49e8-bdfa-c455803db4fc\") " pod="openshift-console/console-f45bd79cb-2gcbm"
Jul 13 18:51:51.677663 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:51.677626    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cd892f91-5591-49e8-bdfa-c455803db4fc-trusted-ca-bundle\") pod \"console-f45bd79cb-2gcbm\" (UID: \"cd892f91-5591-49e8-bdfa-c455803db4fc\") " pod="openshift-console/console-f45bd79cb-2gcbm"
Jul 13 18:51:51.678135 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:51.677746    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cd892f91-5591-49e8-bdfa-c455803db4fc-console-oauth-config\") pod \"console-f45bd79cb-2gcbm\" (UID: \"cd892f91-5591-49e8-bdfa-c455803db4fc\") " pod="openshift-console/console-f45bd79cb-2gcbm"
Jul 13 18:51:51.678135 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:51.677799    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cd892f91-5591-49e8-bdfa-c455803db4fc-oauth-serving-cert\") pod \"console-f45bd79cb-2gcbm\" (UID: \"cd892f91-5591-49e8-bdfa-c455803db4fc\") " pod="openshift-console/console-f45bd79cb-2gcbm"
Jul 13 18:51:51.678135 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:51.677861    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cd892f91-5591-49e8-bdfa-c455803db4fc-service-ca\") pod \"console-f45bd79cb-2gcbm\" (UID: \"cd892f91-5591-49e8-bdfa-c455803db4fc\") " pod="openshift-console/console-f45bd79cb-2gcbm"
Jul 13 18:51:51.678135 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:51.677897    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wcjl2\" (UniqueName: \"kubernetes.io/projected/cd892f91-5591-49e8-bdfa-c455803db4fc-kube-api-access-wcjl2\") pod \"console-f45bd79cb-2gcbm\" (UID: \"cd892f91-5591-49e8-bdfa-c455803db4fc\") " pod="openshift-console/console-f45bd79cb-2gcbm"
Jul 13 18:51:51.678135 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:51.677953    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cd892f91-5591-49e8-bdfa-c455803db4fc-console-config\") pod \"console-f45bd79cb-2gcbm\" (UID: \"cd892f91-5591-49e8-bdfa-c455803db4fc\") " pod="openshift-console/console-f45bd79cb-2gcbm"
Jul 13 18:51:51.779159 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:51.779117    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cd892f91-5591-49e8-bdfa-c455803db4fc-console-serving-cert\") pod \"console-f45bd79cb-2gcbm\" (UID: \"cd892f91-5591-49e8-bdfa-c455803db4fc\") " pod="openshift-console/console-f45bd79cb-2gcbm"
Jul 13 18:51:51.779159 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:51.779165    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cd892f91-5591-49e8-bdfa-c455803db4fc-trusted-ca-bundle\") pod \"console-f45bd79cb-2gcbm\" (UID: \"cd892f91-5591-49e8-bdfa-c455803db4fc\") " pod="openshift-console/console-f45bd79cb-2gcbm"
Jul 13 18:51:51.779393 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:51.779195    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cd892f91-5591-49e8-bdfa-c455803db4fc-console-oauth-config\") pod \"console-f45bd79cb-2gcbm\" (UID: \"cd892f91-5591-49e8-bdfa-c455803db4fc\") " pod="openshift-console/console-f45bd79cb-2gcbm"
Jul 13 18:51:51.779393 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:51.779226    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cd892f91-5591-49e8-bdfa-c455803db4fc-oauth-serving-cert\") pod \"console-f45bd79cb-2gcbm\" (UID: \"cd892f91-5591-49e8-bdfa-c455803db4fc\") " pod="openshift-console/console-f45bd79cb-2gcbm"
Jul 13 18:51:51.779393 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:51.779271    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cd892f91-5591-49e8-bdfa-c455803db4fc-service-ca\") pod \"console-f45bd79cb-2gcbm\" (UID: \"cd892f91-5591-49e8-bdfa-c455803db4fc\") " pod="openshift-console/console-f45bd79cb-2gcbm"
Jul 13 18:51:51.779393 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:51.779298    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wcjl2\" (UniqueName: \"kubernetes.io/projected/cd892f91-5591-49e8-bdfa-c455803db4fc-kube-api-access-wcjl2\") pod \"console-f45bd79cb-2gcbm\" (UID: \"cd892f91-5591-49e8-bdfa-c455803db4fc\") " pod="openshift-console/console-f45bd79cb-2gcbm"
Jul 13 18:51:51.779393 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:51.779344    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cd892f91-5591-49e8-bdfa-c455803db4fc-console-config\") pod \"console-f45bd79cb-2gcbm\" (UID: \"cd892f91-5591-49e8-bdfa-c455803db4fc\") " pod="openshift-console/console-f45bd79cb-2gcbm"
Jul 13 18:51:51.780154 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:51.780099    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cd892f91-5591-49e8-bdfa-c455803db4fc-oauth-serving-cert\") pod \"console-f45bd79cb-2gcbm\" (UID: \"cd892f91-5591-49e8-bdfa-c455803db4fc\") " pod="openshift-console/console-f45bd79cb-2gcbm"
Jul 13 18:51:51.780154 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:51.780130    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cd892f91-5591-49e8-bdfa-c455803db4fc-service-ca\") pod \"console-f45bd79cb-2gcbm\" (UID: \"cd892f91-5591-49e8-bdfa-c455803db4fc\") " pod="openshift-console/console-f45bd79cb-2gcbm"
Jul 13 18:51:51.780339 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:51.780268    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cd892f91-5591-49e8-bdfa-c455803db4fc-trusted-ca-bundle\") pod \"console-f45bd79cb-2gcbm\" (UID: \"cd892f91-5591-49e8-bdfa-c455803db4fc\") " pod="openshift-console/console-f45bd79cb-2gcbm"
Jul 13 18:51:51.780339 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:51.780325    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cd892f91-5591-49e8-bdfa-c455803db4fc-console-config\") pod \"console-f45bd79cb-2gcbm\" (UID: \"cd892f91-5591-49e8-bdfa-c455803db4fc\") " pod="openshift-console/console-f45bd79cb-2gcbm"
Jul 13 18:51:51.781977 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:51.781958    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cd892f91-5591-49e8-bdfa-c455803db4fc-console-serving-cert\") pod \"console-f45bd79cb-2gcbm\" (UID: \"cd892f91-5591-49e8-bdfa-c455803db4fc\") " pod="openshift-console/console-f45bd79cb-2gcbm"
Jul 13 18:51:51.782088 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:51.782014    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cd892f91-5591-49e8-bdfa-c455803db4fc-console-oauth-config\") pod \"console-f45bd79cb-2gcbm\" (UID: \"cd892f91-5591-49e8-bdfa-c455803db4fc\") " pod="openshift-console/console-f45bd79cb-2gcbm"
Jul 13 18:51:51.788190 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:51.788160    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wcjl2\" (UniqueName: \"kubernetes.io/projected/cd892f91-5591-49e8-bdfa-c455803db4fc-kube-api-access-wcjl2\") pod \"console-f45bd79cb-2gcbm\" (UID: \"cd892f91-5591-49e8-bdfa-c455803db4fc\") " pod="openshift-console/console-f45bd79cb-2gcbm"
Jul 13 18:51:51.850105 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:51.850059    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-f45bd79cb-2gcbm"
Jul 13 18:51:52.008626 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:52.008593    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-f45bd79cb-2gcbm"]
Jul 13 18:51:52.009881 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:51:52.009850    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd892f91_5591_49e8_bdfa_c455803db4fc.slice/crio-d7d89f49a7b01b16993045a94d97ce7ebaede4ef595e2b696727371b4c4d8940 WatchSource:0}: Error finding container d7d89f49a7b01b16993045a94d97ce7ebaede4ef595e2b696727371b4c4d8940: Status 404 returned error can't find the container with id d7d89f49a7b01b16993045a94d97ce7ebaede4ef595e2b696727371b4c4d8940
Jul 13 18:51:52.586088 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:52.586051    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-f45bd79cb-2gcbm" event={"ID":"cd892f91-5591-49e8-bdfa-c455803db4fc","Type":"ContainerStarted","Data":"d7d89f49a7b01b16993045a94d97ce7ebaede4ef595e2b696727371b4c4d8940"}
Jul 13 18:51:55.584547 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:55.584519    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-5d9d8df5d-6dwcd"
Jul 13 18:51:56.608430 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:56.608391    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-f45bd79cb-2gcbm" event={"ID":"cd892f91-5591-49e8-bdfa-c455803db4fc","Type":"ContainerStarted","Data":"8c2a54176fe0d846aacec4f7c2b8521b1712c031cc19f2fa0f554e18776e9b03"}
Jul 13 18:51:56.634413 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:51:56.634360    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-f45bd79cb-2gcbm" podStartSLOduration=2.084493546 podStartE2EDuration="5.634339236s" totalImagesPullingTime="3.54984569s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 18:51:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:51:56.632756824 +0000 UTC m=+103.157890268" watchObservedRunningTime="2026-07-13 18:51:56.634339236 +0000 UTC m=+103.159472653"
Jul 13 18:52:01.850774 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:52:01.850726    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-f45bd79cb-2gcbm"
Jul 13 18:52:01.851251 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:52:01.850804    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-f45bd79cb-2gcbm"
Jul 13 18:52:01.855851 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:52:01.855825    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-f45bd79cb-2gcbm"
Jul 13 18:52:02.636259 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:52:02.636229    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-f45bd79cb-2gcbm"
Jul 13 18:52:19.702378 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:52:19.702296    2565 generic.go:356] "Generic (PLEG): container finished" podID="3b153c4b-5864-4143-a644-cb49d7f31e29" containerID="0a9c642bac369bfb4709307fbf3700de0699c1cc76e6a7c2d2ce9d71206ef1e7" exitCode=0
Jul 13 18:52:19.702378 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:52:19.702368    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-2mz4v" event={"ID":"3b153c4b-5864-4143-a644-cb49d7f31e29","Type":"ContainerDied","Data":"0a9c642bac369bfb4709307fbf3700de0699c1cc76e6a7c2d2ce9d71206ef1e7"}
Jul 13 18:52:19.702802 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:52:19.702736    2565 scope.go:117] "RemoveContainer" containerID="0a9c642bac369bfb4709307fbf3700de0699c1cc76e6a7c2d2ce9d71206ef1e7"
Jul 13 18:52:20.708762 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:52:20.708724    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-2mz4v" event={"ID":"3b153c4b-5864-4143-a644-cb49d7f31e29","Type":"ContainerStarted","Data":"b5f9e2d9de860e16efc37a3fbf2a4b35ab265bdd5f352dede4434d84dfe2bc4e"}
Jul 13 18:52:21.030882 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:52:21.030853    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-4rt5g_2ddf2ee1-c9d7-4f30-aa52-b94c6d250c70/serve-healthcheck-canary/0.log"
Jul 13 18:53:03.297745 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:03.297689    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 13 18:53:03.298277 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:03.298022    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="882acd32-9687-4c39-9d84-4893191bccd3" containerName="alertmanager" containerID="cri-o://6bb3111530dc1bb69a0e79a93d26aeb26e6cc826d15707fa661fb541d2e0da61" gracePeriod=120
Jul 13 18:53:03.298277 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:03.298098    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="882acd32-9687-4c39-9d84-4893191bccd3" containerName="kube-rbac-proxy-metric" containerID="cri-o://1c3a82e894ac5dd2f369d0c873cab6ccddb20b91e97a12f953c5a880eddc1583" gracePeriod=120
Jul 13 18:53:03.298277 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:03.298132    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="882acd32-9687-4c39-9d84-4893191bccd3" containerName="prom-label-proxy" containerID="cri-o://a9b0808783015d5baf89d50c38891729d2a6c557adf6316cc933b02b458a2d17" gracePeriod=120
Jul 13 18:53:03.298277 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:03.298184    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="882acd32-9687-4c39-9d84-4893191bccd3" containerName="kube-rbac-proxy" containerID="cri-o://46728ec117baba0f3ce6fcb63da43374adfe6fb5c01227a7600b5d0dba127927" gracePeriod=120
Jul 13 18:53:03.298501 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:03.298146    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="882acd32-9687-4c39-9d84-4893191bccd3" containerName="kube-rbac-proxy-web" containerID="cri-o://445b78855e74fcbcb395888dde2057cf1087d45d9bd8c3066d7af9250ce04570" gracePeriod=120
Jul 13 18:53:03.298501 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:03.298301    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="882acd32-9687-4c39-9d84-4893191bccd3" containerName="config-reloader" containerID="cri-o://d690478c28a2f49e9eecd647956269f98a6b7e7b51316ddff6369cde46c66023" gracePeriod=120
Jul 13 18:53:03.884000 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:03.883967    2565 generic.go:356] "Generic (PLEG): container finished" podID="882acd32-9687-4c39-9d84-4893191bccd3" containerID="a9b0808783015d5baf89d50c38891729d2a6c557adf6316cc933b02b458a2d17" exitCode=0
Jul 13 18:53:03.884000 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:03.883995    2565 generic.go:356] "Generic (PLEG): container finished" podID="882acd32-9687-4c39-9d84-4893191bccd3" containerID="46728ec117baba0f3ce6fcb63da43374adfe6fb5c01227a7600b5d0dba127927" exitCode=0
Jul 13 18:53:03.884000 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:03.884003    2565 generic.go:356] "Generic (PLEG): container finished" podID="882acd32-9687-4c39-9d84-4893191bccd3" containerID="d690478c28a2f49e9eecd647956269f98a6b7e7b51316ddff6369cde46c66023" exitCode=0
Jul 13 18:53:03.884000 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:03.884009    2565 generic.go:356] "Generic (PLEG): container finished" podID="882acd32-9687-4c39-9d84-4893191bccd3" containerID="6bb3111530dc1bb69a0e79a93d26aeb26e6cc826d15707fa661fb541d2e0da61" exitCode=0
Jul 13 18:53:03.884269 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:03.884041    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"882acd32-9687-4c39-9d84-4893191bccd3","Type":"ContainerDied","Data":"a9b0808783015d5baf89d50c38891729d2a6c557adf6316cc933b02b458a2d17"}
Jul 13 18:53:03.884269 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:03.884109    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"882acd32-9687-4c39-9d84-4893191bccd3","Type":"ContainerDied","Data":"46728ec117baba0f3ce6fcb63da43374adfe6fb5c01227a7600b5d0dba127927"}
Jul 13 18:53:03.884269 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:03.884136    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"882acd32-9687-4c39-9d84-4893191bccd3","Type":"ContainerDied","Data":"d690478c28a2f49e9eecd647956269f98a6b7e7b51316ddff6369cde46c66023"}
Jul 13 18:53:03.884269 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:03.884161    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"882acd32-9687-4c39-9d84-4893191bccd3","Type":"ContainerDied","Data":"6bb3111530dc1bb69a0e79a93d26aeb26e6cc826d15707fa661fb541d2e0da61"}
Jul 13 18:53:04.544751 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:04.544724    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 13 18:53:04.615552 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:04.615467    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/882acd32-9687-4c39-9d84-4893191bccd3-config-out\") pod \"882acd32-9687-4c39-9d84-4893191bccd3\" (UID: \"882acd32-9687-4c39-9d84-4893191bccd3\") "
Jul 13 18:53:04.615552 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:04.615546    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/882acd32-9687-4c39-9d84-4893191bccd3-web-config\") pod \"882acd32-9687-4c39-9d84-4893191bccd3\" (UID: \"882acd32-9687-4c39-9d84-4893191bccd3\") "
Jul 13 18:53:04.615799 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:04.615575    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/882acd32-9687-4c39-9d84-4893191bccd3-cluster-tls-config\") pod \"882acd32-9687-4c39-9d84-4893191bccd3\" (UID: \"882acd32-9687-4c39-9d84-4893191bccd3\") "
Jul 13 18:53:04.615799 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:04.615615    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/882acd32-9687-4c39-9d84-4893191bccd3-secret-alertmanager-kube-rbac-proxy\") pod \"882acd32-9687-4c39-9d84-4893191bccd3\" (UID: \"882acd32-9687-4c39-9d84-4893191bccd3\") "
Jul 13 18:53:04.615799 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:04.615647    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/882acd32-9687-4c39-9d84-4893191bccd3-secret-alertmanager-kube-rbac-proxy-metric\") pod \"882acd32-9687-4c39-9d84-4893191bccd3\" (UID: \"882acd32-9687-4c39-9d84-4893191bccd3\") "
Jul 13 18:53:04.615799 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:04.615673    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/882acd32-9687-4c39-9d84-4893191bccd3-secret-alertmanager-main-tls\") pod \"882acd32-9687-4c39-9d84-4893191bccd3\" (UID: \"882acd32-9687-4c39-9d84-4893191bccd3\") "
Jul 13 18:53:04.617483 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:04.616129    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/882acd32-9687-4c39-9d84-4893191bccd3-tls-assets\") pod \"882acd32-9687-4c39-9d84-4893191bccd3\" (UID: \"882acd32-9687-4c39-9d84-4893191bccd3\") "
Jul 13 18:53:04.617483 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:04.616171    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/882acd32-9687-4c39-9d84-4893191bccd3-config-volume\") pod \"882acd32-9687-4c39-9d84-4893191bccd3\" (UID: \"882acd32-9687-4c39-9d84-4893191bccd3\") "
Jul 13 18:53:04.617483 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:04.616270    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/882acd32-9687-4c39-9d84-4893191bccd3-secret-alertmanager-kube-rbac-proxy-web\") pod \"882acd32-9687-4c39-9d84-4893191bccd3\" (UID: \"882acd32-9687-4c39-9d84-4893191bccd3\") "
Jul 13 18:53:04.617483 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:04.616441    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/882acd32-9687-4c39-9d84-4893191bccd3-alertmanager-main-db\") pod \"882acd32-9687-4c39-9d84-4893191bccd3\" (UID: \"882acd32-9687-4c39-9d84-4893191bccd3\") "
Jul 13 18:53:04.617483 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:04.616485    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tds24\" (UniqueName: \"kubernetes.io/projected/882acd32-9687-4c39-9d84-4893191bccd3-kube-api-access-tds24\") pod \"882acd32-9687-4c39-9d84-4893191bccd3\" (UID: \"882acd32-9687-4c39-9d84-4893191bccd3\") "
Jul 13 18:53:04.617483 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:04.616514    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/882acd32-9687-4c39-9d84-4893191bccd3-metrics-client-ca\") pod \"882acd32-9687-4c39-9d84-4893191bccd3\" (UID: \"882acd32-9687-4c39-9d84-4893191bccd3\") "
Jul 13 18:53:04.617483 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:04.616549    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/882acd32-9687-4c39-9d84-4893191bccd3-alertmanager-trusted-ca-bundle\") pod \"882acd32-9687-4c39-9d84-4893191bccd3\" (UID: \"882acd32-9687-4c39-9d84-4893191bccd3\") "
Jul 13 18:53:04.617483 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:04.617044    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/882acd32-9687-4c39-9d84-4893191bccd3-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "882acd32-9687-4c39-9d84-4893191bccd3" (UID: "882acd32-9687-4c39-9d84-4893191bccd3"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 18:53:04.617483 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:04.617220    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/882acd32-9687-4c39-9d84-4893191bccd3-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "882acd32-9687-4c39-9d84-4893191bccd3" (UID: "882acd32-9687-4c39-9d84-4893191bccd3"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 18:53:04.617483 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:04.617444    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/882acd32-9687-4c39-9d84-4893191bccd3-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "882acd32-9687-4c39-9d84-4893191bccd3" (UID: "882acd32-9687-4c39-9d84-4893191bccd3"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 18:53:04.618539 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:04.618512    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/882acd32-9687-4c39-9d84-4893191bccd3-config-out" (OuterVolumeSpecName: "config-out") pod "882acd32-9687-4c39-9d84-4893191bccd3" (UID: "882acd32-9687-4c39-9d84-4893191bccd3"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 18:53:04.618939 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:04.618892    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/882acd32-9687-4c39-9d84-4893191bccd3-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "882acd32-9687-4c39-9d84-4893191bccd3" (UID: "882acd32-9687-4c39-9d84-4893191bccd3"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 18:53:04.619105 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:04.619073    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/882acd32-9687-4c39-9d84-4893191bccd3-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "882acd32-9687-4c39-9d84-4893191bccd3" (UID: "882acd32-9687-4c39-9d84-4893191bccd3"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 18:53:04.619642 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:04.619608    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/882acd32-9687-4c39-9d84-4893191bccd3-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "882acd32-9687-4c39-9d84-4893191bccd3" (UID: "882acd32-9687-4c39-9d84-4893191bccd3"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 18:53:04.620640 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:04.620596    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/882acd32-9687-4c39-9d84-4893191bccd3-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "882acd32-9687-4c39-9d84-4893191bccd3" (UID: "882acd32-9687-4c39-9d84-4893191bccd3"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 18:53:04.620640 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:04.620618    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/882acd32-9687-4c39-9d84-4893191bccd3-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "882acd32-9687-4c39-9d84-4893191bccd3" (UID: "882acd32-9687-4c39-9d84-4893191bccd3"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 18:53:04.621227 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:04.621202    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/882acd32-9687-4c39-9d84-4893191bccd3-config-volume" (OuterVolumeSpecName: "config-volume") pod "882acd32-9687-4c39-9d84-4893191bccd3" (UID: "882acd32-9687-4c39-9d84-4893191bccd3"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 18:53:04.622011 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:04.621986    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/882acd32-9687-4c39-9d84-4893191bccd3-kube-api-access-tds24" (OuterVolumeSpecName: "kube-api-access-tds24") pod "882acd32-9687-4c39-9d84-4893191bccd3" (UID: "882acd32-9687-4c39-9d84-4893191bccd3"). InnerVolumeSpecName "kube-api-access-tds24". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 18:53:04.624671 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:04.624506    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/882acd32-9687-4c39-9d84-4893191bccd3-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "882acd32-9687-4c39-9d84-4893191bccd3" (UID: "882acd32-9687-4c39-9d84-4893191bccd3"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 18:53:04.631889 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:04.631867    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/882acd32-9687-4c39-9d84-4893191bccd3-web-config" (OuterVolumeSpecName: "web-config") pod "882acd32-9687-4c39-9d84-4893191bccd3" (UID: "882acd32-9687-4c39-9d84-4893191bccd3"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 18:53:04.717161 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:04.717118    2565 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/882acd32-9687-4c39-9d84-4893191bccd3-web-config\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 13 18:53:04.717161 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:04.717152    2565 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/882acd32-9687-4c39-9d84-4893191bccd3-cluster-tls-config\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 13 18:53:04.717161 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:04.717163    2565 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/882acd32-9687-4c39-9d84-4893191bccd3-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 13 18:53:04.717385 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:04.717174    2565 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/882acd32-9687-4c39-9d84-4893191bccd3-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 13 18:53:04.717385 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:04.717183    2565 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/882acd32-9687-4c39-9d84-4893191bccd3-secret-alertmanager-main-tls\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 13 18:53:04.717385 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:04.717192    2565 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/882acd32-9687-4c39-9d84-4893191bccd3-tls-assets\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 13 18:53:04.717385 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:04.717203    2565 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/882acd32-9687-4c39-9d84-4893191bccd3-config-volume\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 13 18:53:04.717385 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:04.717210    2565 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/882acd32-9687-4c39-9d84-4893191bccd3-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 13 18:53:04.717385 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:04.717219    2565 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/882acd32-9687-4c39-9d84-4893191bccd3-alertmanager-main-db\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 13 18:53:04.717385 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:04.717227    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tds24\" (UniqueName: \"kubernetes.io/projected/882acd32-9687-4c39-9d84-4893191bccd3-kube-api-access-tds24\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 13 18:53:04.717385 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:04.717235    2565 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/882acd32-9687-4c39-9d84-4893191bccd3-metrics-client-ca\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 13 18:53:04.717385 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:04.717243    2565 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/882acd32-9687-4c39-9d84-4893191bccd3-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 13 18:53:04.717385 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:04.717252    2565 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/882acd32-9687-4c39-9d84-4893191bccd3-config-out\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 13 18:53:04.890627 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:04.890527    2565 generic.go:356] "Generic (PLEG): container finished" podID="882acd32-9687-4c39-9d84-4893191bccd3" containerID="1c3a82e894ac5dd2f369d0c873cab6ccddb20b91e97a12f953c5a880eddc1583" exitCode=0
Jul 13 18:53:04.890627 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:04.890559    2565 generic.go:356] "Generic (PLEG): container finished" podID="882acd32-9687-4c39-9d84-4893191bccd3" containerID="445b78855e74fcbcb395888dde2057cf1087d45d9bd8c3066d7af9250ce04570" exitCode=0
Jul 13 18:53:04.890627 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:04.890607    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"882acd32-9687-4c39-9d84-4893191bccd3","Type":"ContainerDied","Data":"1c3a82e894ac5dd2f369d0c873cab6ccddb20b91e97a12f953c5a880eddc1583"}
Jul 13 18:53:04.890844 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:04.890627    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 13 18:53:04.890844 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:04.890721    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"882acd32-9687-4c39-9d84-4893191bccd3","Type":"ContainerDied","Data":"445b78855e74fcbcb395888dde2057cf1087d45d9bd8c3066d7af9250ce04570"}
Jul 13 18:53:04.890844 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:04.890753    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"882acd32-9687-4c39-9d84-4893191bccd3","Type":"ContainerDied","Data":"44d22af040c71313966e6c708827472cb2ab28f5cb7c38a3f3d6c1d31f38f0fe"}
Jul 13 18:53:04.890844 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:04.890793    2565 scope.go:117] "RemoveContainer" containerID="a9b0808783015d5baf89d50c38891729d2a6c557adf6316cc933b02b458a2d17"
Jul 13 18:53:04.899694 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:04.899678    2565 scope.go:117] "RemoveContainer" containerID="1c3a82e894ac5dd2f369d0c873cab6ccddb20b91e97a12f953c5a880eddc1583"
Jul 13 18:53:04.906729 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:04.906691    2565 scope.go:117] "RemoveContainer" containerID="46728ec117baba0f3ce6fcb63da43374adfe6fb5c01227a7600b5d0dba127927"
Jul 13 18:53:04.913489 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:04.913472    2565 scope.go:117] "RemoveContainer" containerID="445b78855e74fcbcb395888dde2057cf1087d45d9bd8c3066d7af9250ce04570"
Jul 13 18:53:04.917376 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:04.917358    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 13 18:53:04.921031 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:04.921013    2565 scope.go:117] "RemoveContainer" containerID="d690478c28a2f49e9eecd647956269f98a6b7e7b51316ddff6369cde46c66023"
Jul 13 18:53:04.922716 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:04.922685    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 13 18:53:04.928374 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:04.928356    2565 scope.go:117] "RemoveContainer" containerID="6bb3111530dc1bb69a0e79a93d26aeb26e6cc826d15707fa661fb541d2e0da61"
Jul 13 18:53:04.935762 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:04.935740    2565 scope.go:117] "RemoveContainer" containerID="0447498ed817a0745ac7f850e776da80aa7c6f89bed915fc236d39ceca577ddf"
Jul 13 18:53:04.942953 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:04.942925    2565 scope.go:117] "RemoveContainer" containerID="a9b0808783015d5baf89d50c38891729d2a6c557adf6316cc933b02b458a2d17"
Jul 13 18:53:04.943212 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:53:04.943192    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9b0808783015d5baf89d50c38891729d2a6c557adf6316cc933b02b458a2d17\": container with ID starting with a9b0808783015d5baf89d50c38891729d2a6c557adf6316cc933b02b458a2d17 not found: ID does not exist" containerID="a9b0808783015d5baf89d50c38891729d2a6c557adf6316cc933b02b458a2d17"
Jul 13 18:53:04.943265 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:04.943221    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9b0808783015d5baf89d50c38891729d2a6c557adf6316cc933b02b458a2d17"} err="failed to get container status \"a9b0808783015d5baf89d50c38891729d2a6c557adf6316cc933b02b458a2d17\": rpc error: code = NotFound desc = could not find container \"a9b0808783015d5baf89d50c38891729d2a6c557adf6316cc933b02b458a2d17\": container with ID starting with a9b0808783015d5baf89d50c38891729d2a6c557adf6316cc933b02b458a2d17 not found: ID does not exist"
Jul 13 18:53:04.943265 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:04.943256    2565 scope.go:117] "RemoveContainer" containerID="1c3a82e894ac5dd2f369d0c873cab6ccddb20b91e97a12f953c5a880eddc1583"
Jul 13 18:53:04.943482 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:53:04.943466    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c3a82e894ac5dd2f369d0c873cab6ccddb20b91e97a12f953c5a880eddc1583\": container with ID starting with 1c3a82e894ac5dd2f369d0c873cab6ccddb20b91e97a12f953c5a880eddc1583 not found: ID does not exist" containerID="1c3a82e894ac5dd2f369d0c873cab6ccddb20b91e97a12f953c5a880eddc1583"
Jul 13 18:53:04.943522 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:04.943487    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c3a82e894ac5dd2f369d0c873cab6ccddb20b91e97a12f953c5a880eddc1583"} err="failed to get container status \"1c3a82e894ac5dd2f369d0c873cab6ccddb20b91e97a12f953c5a880eddc1583\": rpc error: code = NotFound desc = could not find container \"1c3a82e894ac5dd2f369d0c873cab6ccddb20b91e97a12f953c5a880eddc1583\": container with ID starting with 1c3a82e894ac5dd2f369d0c873cab6ccddb20b91e97a12f953c5a880eddc1583 not found: ID does not exist"
Jul 13 18:53:04.943522 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:04.943501    2565 scope.go:117] "RemoveContainer" containerID="46728ec117baba0f3ce6fcb63da43374adfe6fb5c01227a7600b5d0dba127927"
Jul 13 18:53:04.943752 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:53:04.943736    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46728ec117baba0f3ce6fcb63da43374adfe6fb5c01227a7600b5d0dba127927\": container with ID starting with 46728ec117baba0f3ce6fcb63da43374adfe6fb5c01227a7600b5d0dba127927 not found: ID does not exist" containerID="46728ec117baba0f3ce6fcb63da43374adfe6fb5c01227a7600b5d0dba127927"
Jul 13 18:53:04.943812 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:04.943759    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46728ec117baba0f3ce6fcb63da43374adfe6fb5c01227a7600b5d0dba127927"} err="failed to get container status \"46728ec117baba0f3ce6fcb63da43374adfe6fb5c01227a7600b5d0dba127927\": rpc error: code = NotFound desc = could not find container \"46728ec117baba0f3ce6fcb63da43374adfe6fb5c01227a7600b5d0dba127927\": container with ID starting with 46728ec117baba0f3ce6fcb63da43374adfe6fb5c01227a7600b5d0dba127927 not found: ID does not exist"
Jul 13 18:53:04.943812 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:04.943775    2565 scope.go:117] "RemoveContainer" containerID="445b78855e74fcbcb395888dde2057cf1087d45d9bd8c3066d7af9250ce04570"
Jul 13 18:53:04.943996 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:53:04.943982    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"445b78855e74fcbcb395888dde2057cf1087d45d9bd8c3066d7af9250ce04570\": container with ID starting with 445b78855e74fcbcb395888dde2057cf1087d45d9bd8c3066d7af9250ce04570 not found: ID does not exist" containerID="445b78855e74fcbcb395888dde2057cf1087d45d9bd8c3066d7af9250ce04570"
Jul 13 18:53:04.944032 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:04.943999    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"445b78855e74fcbcb395888dde2057cf1087d45d9bd8c3066d7af9250ce04570"} err="failed to get container status \"445b78855e74fcbcb395888dde2057cf1087d45d9bd8c3066d7af9250ce04570\": rpc error: code = NotFound desc = could not find container \"445b78855e74fcbcb395888dde2057cf1087d45d9bd8c3066d7af9250ce04570\": container with ID starting with 445b78855e74fcbcb395888dde2057cf1087d45d9bd8c3066d7af9250ce04570 not found: ID does not exist"
Jul 13 18:53:04.944032 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:04.944019    2565 scope.go:117] "RemoveContainer" containerID="d690478c28a2f49e9eecd647956269f98a6b7e7b51316ddff6369cde46c66023"
Jul 13 18:53:04.944253 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:53:04.944240    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d690478c28a2f49e9eecd647956269f98a6b7e7b51316ddff6369cde46c66023\": container with ID starting with d690478c28a2f49e9eecd647956269f98a6b7e7b51316ddff6369cde46c66023 not found: ID does not exist" containerID="d690478c28a2f49e9eecd647956269f98a6b7e7b51316ddff6369cde46c66023"
Jul 13 18:53:04.944292 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:04.944256    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d690478c28a2f49e9eecd647956269f98a6b7e7b51316ddff6369cde46c66023"} err="failed to get container status \"d690478c28a2f49e9eecd647956269f98a6b7e7b51316ddff6369cde46c66023\": rpc error: code = NotFound desc = could not find container \"d690478c28a2f49e9eecd647956269f98a6b7e7b51316ddff6369cde46c66023\": container with ID starting with d690478c28a2f49e9eecd647956269f98a6b7e7b51316ddff6369cde46c66023 not found: ID does not exist"
Jul 13 18:53:04.944292 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:04.944272    2565 scope.go:117] "RemoveContainer" containerID="6bb3111530dc1bb69a0e79a93d26aeb26e6cc826d15707fa661fb541d2e0da61"
Jul 13 18:53:04.944508 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:53:04.944490    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6bb3111530dc1bb69a0e79a93d26aeb26e6cc826d15707fa661fb541d2e0da61\": container with ID starting with 6bb3111530dc1bb69a0e79a93d26aeb26e6cc826d15707fa661fb541d2e0da61 not found: ID does not exist" containerID="6bb3111530dc1bb69a0e79a93d26aeb26e6cc826d15707fa661fb541d2e0da61"
Jul 13 18:53:04.944558 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:04.944524    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6bb3111530dc1bb69a0e79a93d26aeb26e6cc826d15707fa661fb541d2e0da61"} err="failed to get container status \"6bb3111530dc1bb69a0e79a93d26aeb26e6cc826d15707fa661fb541d2e0da61\": rpc error: code = NotFound desc = could not find container \"6bb3111530dc1bb69a0e79a93d26aeb26e6cc826d15707fa661fb541d2e0da61\": container with ID starting with 6bb3111530dc1bb69a0e79a93d26aeb26e6cc826d15707fa661fb541d2e0da61 not found: ID does not exist"
Jul 13 18:53:04.944558 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:04.944550    2565 scope.go:117] "RemoveContainer" containerID="0447498ed817a0745ac7f850e776da80aa7c6f89bed915fc236d39ceca577ddf"
Jul 13 18:53:04.944765 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:53:04.944751    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0447498ed817a0745ac7f850e776da80aa7c6f89bed915fc236d39ceca577ddf\": container with ID starting with 0447498ed817a0745ac7f850e776da80aa7c6f89bed915fc236d39ceca577ddf not found: ID does not exist" containerID="0447498ed817a0745ac7f850e776da80aa7c6f89bed915fc236d39ceca577ddf"
Jul 13 18:53:04.944816 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:04.944769    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0447498ed817a0745ac7f850e776da80aa7c6f89bed915fc236d39ceca577ddf"} err="failed to get container status \"0447498ed817a0745ac7f850e776da80aa7c6f89bed915fc236d39ceca577ddf\": rpc error: code = NotFound desc = could not find container \"0447498ed817a0745ac7f850e776da80aa7c6f89bed915fc236d39ceca577ddf\": container with ID starting with 0447498ed817a0745ac7f850e776da80aa7c6f89bed915fc236d39ceca577ddf not found: ID does not exist"
Jul 13 18:53:04.944816 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:04.944781    2565 scope.go:117] "RemoveContainer" containerID="a9b0808783015d5baf89d50c38891729d2a6c557adf6316cc933b02b458a2d17"
Jul 13 18:53:04.944985 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:04.944968    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9b0808783015d5baf89d50c38891729d2a6c557adf6316cc933b02b458a2d17"} err="failed to get container status \"a9b0808783015d5baf89d50c38891729d2a6c557adf6316cc933b02b458a2d17\": rpc error: code = NotFound desc = could not find container \"a9b0808783015d5baf89d50c38891729d2a6c557adf6316cc933b02b458a2d17\": container with ID starting with a9b0808783015d5baf89d50c38891729d2a6c557adf6316cc933b02b458a2d17 not found: ID does not exist"
Jul 13 18:53:04.945043 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:04.944985    2565 scope.go:117] "RemoveContainer" containerID="1c3a82e894ac5dd2f369d0c873cab6ccddb20b91e97a12f953c5a880eddc1583"
Jul 13 18:53:04.945203 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:04.945185    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c3a82e894ac5dd2f369d0c873cab6ccddb20b91e97a12f953c5a880eddc1583"} err="failed to get container status \"1c3a82e894ac5dd2f369d0c873cab6ccddb20b91e97a12f953c5a880eddc1583\": rpc error: code = NotFound desc = could not find container \"1c3a82e894ac5dd2f369d0c873cab6ccddb20b91e97a12f953c5a880eddc1583\": container with ID starting with 1c3a82e894ac5dd2f369d0c873cab6ccddb20b91e97a12f953c5a880eddc1583 not found: ID does not exist"
Jul 13 18:53:04.945244 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:04.945205    2565 scope.go:117] "RemoveContainer" containerID="46728ec117baba0f3ce6fcb63da43374adfe6fb5c01227a7600b5d0dba127927"
Jul 13 18:53:04.945389 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:04.945373    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46728ec117baba0f3ce6fcb63da43374adfe6fb5c01227a7600b5d0dba127927"} err="failed to get container status \"46728ec117baba0f3ce6fcb63da43374adfe6fb5c01227a7600b5d0dba127927\": rpc error: code = NotFound desc = could not find container \"46728ec117baba0f3ce6fcb63da43374adfe6fb5c01227a7600b5d0dba127927\": container with ID starting with 46728ec117baba0f3ce6fcb63da43374adfe6fb5c01227a7600b5d0dba127927 not found: ID does not exist"
Jul 13 18:53:04.945425 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:04.945389    2565 scope.go:117] "RemoveContainer" containerID="445b78855e74fcbcb395888dde2057cf1087d45d9bd8c3066d7af9250ce04570"
Jul 13 18:53:04.945577 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:04.945561    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"445b78855e74fcbcb395888dde2057cf1087d45d9bd8c3066d7af9250ce04570"} err="failed to get container status \"445b78855e74fcbcb395888dde2057cf1087d45d9bd8c3066d7af9250ce04570\": rpc error: code = NotFound desc = could not find container \"445b78855e74fcbcb395888dde2057cf1087d45d9bd8c3066d7af9250ce04570\": container with ID starting with 445b78855e74fcbcb395888dde2057cf1087d45d9bd8c3066d7af9250ce04570 not found: ID does not exist"
Jul 13 18:53:04.945615 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:04.945578    2565 scope.go:117] "RemoveContainer" containerID="d690478c28a2f49e9eecd647956269f98a6b7e7b51316ddff6369cde46c66023"
Jul 13 18:53:04.945794 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:04.945779    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d690478c28a2f49e9eecd647956269f98a6b7e7b51316ddff6369cde46c66023"} err="failed to get container status \"d690478c28a2f49e9eecd647956269f98a6b7e7b51316ddff6369cde46c66023\": rpc error: code = NotFound desc = could not find container \"d690478c28a2f49e9eecd647956269f98a6b7e7b51316ddff6369cde46c66023\": container with ID starting with d690478c28a2f49e9eecd647956269f98a6b7e7b51316ddff6369cde46c66023 not found: ID does not exist"
Jul 13 18:53:04.945837 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:04.945794    2565 scope.go:117] "RemoveContainer" containerID="6bb3111530dc1bb69a0e79a93d26aeb26e6cc826d15707fa661fb541d2e0da61"
Jul 13 18:53:04.946010 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:04.945995    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6bb3111530dc1bb69a0e79a93d26aeb26e6cc826d15707fa661fb541d2e0da61"} err="failed to get container status \"6bb3111530dc1bb69a0e79a93d26aeb26e6cc826d15707fa661fb541d2e0da61\": rpc error: code = NotFound desc = could not find container \"6bb3111530dc1bb69a0e79a93d26aeb26e6cc826d15707fa661fb541d2e0da61\": container with ID starting with 6bb3111530dc1bb69a0e79a93d26aeb26e6cc826d15707fa661fb541d2e0da61 not found: ID does not exist"
Jul 13 18:53:04.946057 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:04.946019    2565 scope.go:117] "RemoveContainer" containerID="0447498ed817a0745ac7f850e776da80aa7c6f89bed915fc236d39ceca577ddf"
Jul 13 18:53:04.946255 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:04.946231    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0447498ed817a0745ac7f850e776da80aa7c6f89bed915fc236d39ceca577ddf"} err="failed to get container status \"0447498ed817a0745ac7f850e776da80aa7c6f89bed915fc236d39ceca577ddf\": rpc error: code = NotFound desc = could not find container \"0447498ed817a0745ac7f850e776da80aa7c6f89bed915fc236d39ceca577ddf\": container with ID starting with 0447498ed817a0745ac7f850e776da80aa7c6f89bed915fc236d39ceca577ddf not found: ID does not exist"
Jul 13 18:53:04.954172 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:04.954152    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 13 18:53:04.954453 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:04.954442    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="882acd32-9687-4c39-9d84-4893191bccd3" containerName="prom-label-proxy"
Jul 13 18:53:04.954500 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:04.954455    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="882acd32-9687-4c39-9d84-4893191bccd3" containerName="prom-label-proxy"
Jul 13 18:53:04.954500 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:04.954466    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="882acd32-9687-4c39-9d84-4893191bccd3" containerName="kube-rbac-proxy"
Jul 13 18:53:04.954500 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:04.954472    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="882acd32-9687-4c39-9d84-4893191bccd3" containerName="kube-rbac-proxy"
Jul 13 18:53:04.954500 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:04.954483    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="882acd32-9687-4c39-9d84-4893191bccd3" containerName="config-reloader"
Jul 13 18:53:04.954500 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:04.954488    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="882acd32-9687-4c39-9d84-4893191bccd3" containerName="config-reloader"
Jul 13 18:53:04.954500 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:04.954499    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="882acd32-9687-4c39-9d84-4893191bccd3" containerName="init-config-reloader"
Jul 13 18:53:04.954732 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:04.954505    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="882acd32-9687-4c39-9d84-4893191bccd3" containerName="init-config-reloader"
Jul 13 18:53:04.954732 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:04.954513    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="882acd32-9687-4c39-9d84-4893191bccd3" containerName="kube-rbac-proxy-web"
Jul 13 18:53:04.954732 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:04.954518    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="882acd32-9687-4c39-9d84-4893191bccd3" containerName="kube-rbac-proxy-web"
Jul 13 18:53:04.954732 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:04.954526    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="882acd32-9687-4c39-9d84-4893191bccd3" containerName="alertmanager"
Jul 13 18:53:04.954732 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:04.954530    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="882acd32-9687-4c39-9d84-4893191bccd3" containerName="alertmanager"
Jul 13 18:53:04.954732 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:04.954537    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="882acd32-9687-4c39-9d84-4893191bccd3" containerName="kube-rbac-proxy-metric"
Jul 13 18:53:04.954732 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:04.954543    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="882acd32-9687-4c39-9d84-4893191bccd3" containerName="kube-rbac-proxy-metric"
Jul 13 18:53:04.954732 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:04.954597    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="882acd32-9687-4c39-9d84-4893191bccd3" containerName="alertmanager"
Jul 13 18:53:04.954732 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:04.954605    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="882acd32-9687-4c39-9d84-4893191bccd3" containerName="kube-rbac-proxy-web"
Jul 13 18:53:04.954732 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:04.954611    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="882acd32-9687-4c39-9d84-4893191bccd3" containerName="prom-label-proxy"
Jul 13 18:53:04.954732 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:04.954617    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="882acd32-9687-4c39-9d84-4893191bccd3" containerName="kube-rbac-proxy-metric"
Jul 13 18:53:04.954732 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:04.954624    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="882acd32-9687-4c39-9d84-4893191bccd3" containerName="config-reloader"
Jul 13 18:53:04.954732 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:04.954629    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="882acd32-9687-4c39-9d84-4893191bccd3" containerName="kube-rbac-proxy"
Jul 13 18:53:04.959804 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:04.959788    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 13 18:53:04.964069 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:04.962719    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Jul 13 18:53:04.964069 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:04.962952    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Jul 13 18:53:04.964069 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:04.963214    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Jul 13 18:53:04.964069 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:04.963377    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Jul 13 18:53:04.964069 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:04.963560    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Jul 13 18:53:04.964069 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:04.963760    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Jul 13 18:53:04.964069 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:04.963936    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Jul 13 18:53:04.964467 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:04.964301    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Jul 13 18:53:04.964662 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:04.964643    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-m2cff\""
Jul 13 18:53:04.969679 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:04.969661    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Jul 13 18:53:04.976001 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:04.975981    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 13 18:53:05.019759 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:05.019729    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a22aa65f-fb55-49ec-a10e-c14a494422de-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"a22aa65f-fb55-49ec-a10e-c14a494422de\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 13 18:53:05.019759 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:05.019761    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a22aa65f-fb55-49ec-a10e-c14a494422de-config-out\") pod \"alertmanager-main-0\" (UID: \"a22aa65f-fb55-49ec-a10e-c14a494422de\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 13 18:53:05.019971 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:05.019783    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a22aa65f-fb55-49ec-a10e-c14a494422de-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"a22aa65f-fb55-49ec-a10e-c14a494422de\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 13 18:53:05.019971 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:05.019798    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvclf\" (UniqueName: \"kubernetes.io/projected/a22aa65f-fb55-49ec-a10e-c14a494422de-kube-api-access-nvclf\") pod \"alertmanager-main-0\" (UID: \"a22aa65f-fb55-49ec-a10e-c14a494422de\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 13 18:53:05.019971 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:05.019895    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a22aa65f-fb55-49ec-a10e-c14a494422de-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"a22aa65f-fb55-49ec-a10e-c14a494422de\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 13 18:53:05.019971 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:05.019945    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a22aa65f-fb55-49ec-a10e-c14a494422de-web-config\") pod \"alertmanager-main-0\" (UID: \"a22aa65f-fb55-49ec-a10e-c14a494422de\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 13 18:53:05.020166 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:05.020022    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a22aa65f-fb55-49ec-a10e-c14a494422de-config-volume\") pod \"alertmanager-main-0\" (UID: \"a22aa65f-fb55-49ec-a10e-c14a494422de\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 13 18:53:05.020166 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:05.020053    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a22aa65f-fb55-49ec-a10e-c14a494422de-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"a22aa65f-fb55-49ec-a10e-c14a494422de\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 13 18:53:05.020166 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:05.020086    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a22aa65f-fb55-49ec-a10e-c14a494422de-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"a22aa65f-fb55-49ec-a10e-c14a494422de\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 13 18:53:05.020166 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:05.020123    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a22aa65f-fb55-49ec-a10e-c14a494422de-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"a22aa65f-fb55-49ec-a10e-c14a494422de\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 13 18:53:05.020166 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:05.020151    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a22aa65f-fb55-49ec-a10e-c14a494422de-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"a22aa65f-fb55-49ec-a10e-c14a494422de\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 13 18:53:05.020347 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:05.020187    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a22aa65f-fb55-49ec-a10e-c14a494422de-tls-assets\") pod \"alertmanager-main-0\" (UID: \"a22aa65f-fb55-49ec-a10e-c14a494422de\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 13 18:53:05.020347 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:05.020215    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a22aa65f-fb55-49ec-a10e-c14a494422de-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"a22aa65f-fb55-49ec-a10e-c14a494422de\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 13 18:53:05.121022 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:05.120980    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a22aa65f-fb55-49ec-a10e-c14a494422de-config-volume\") pod \"alertmanager-main-0\" (UID: \"a22aa65f-fb55-49ec-a10e-c14a494422de\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 13 18:53:05.121022 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:05.121020    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a22aa65f-fb55-49ec-a10e-c14a494422de-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"a22aa65f-fb55-49ec-a10e-c14a494422de\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 13 18:53:05.121282 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:05.121046    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a22aa65f-fb55-49ec-a10e-c14a494422de-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"a22aa65f-fb55-49ec-a10e-c14a494422de\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 13 18:53:05.121282 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:05.121068    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a22aa65f-fb55-49ec-a10e-c14a494422de-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"a22aa65f-fb55-49ec-a10e-c14a494422de\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 13 18:53:05.121282 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:05.121084    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a22aa65f-fb55-49ec-a10e-c14a494422de-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"a22aa65f-fb55-49ec-a10e-c14a494422de\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 13 18:53:05.121282 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:05.121105    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a22aa65f-fb55-49ec-a10e-c14a494422de-tls-assets\") pod \"alertmanager-main-0\" (UID: \"a22aa65f-fb55-49ec-a10e-c14a494422de\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 13 18:53:05.121282 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:05.121129    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a22aa65f-fb55-49ec-a10e-c14a494422de-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"a22aa65f-fb55-49ec-a10e-c14a494422de\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 13 18:53:05.121282 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:05.121153    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a22aa65f-fb55-49ec-a10e-c14a494422de-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"a22aa65f-fb55-49ec-a10e-c14a494422de\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 13 18:53:05.121282 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:05.121169    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a22aa65f-fb55-49ec-a10e-c14a494422de-config-out\") pod \"alertmanager-main-0\" (UID: \"a22aa65f-fb55-49ec-a10e-c14a494422de\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 13 18:53:05.121282 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:05.121201    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a22aa65f-fb55-49ec-a10e-c14a494422de-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"a22aa65f-fb55-49ec-a10e-c14a494422de\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 13 18:53:05.121282 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:05.121223    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvclf\" (UniqueName: \"kubernetes.io/projected/a22aa65f-fb55-49ec-a10e-c14a494422de-kube-api-access-nvclf\") pod \"alertmanager-main-0\" (UID: \"a22aa65f-fb55-49ec-a10e-c14a494422de\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 13 18:53:05.121282 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:05.121282    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a22aa65f-fb55-49ec-a10e-c14a494422de-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"a22aa65f-fb55-49ec-a10e-c14a494422de\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 13 18:53:05.121687 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:05.121308    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a22aa65f-fb55-49ec-a10e-c14a494422de-web-config\") pod \"alertmanager-main-0\" (UID: \"a22aa65f-fb55-49ec-a10e-c14a494422de\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 13 18:53:05.121687 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:05.121526    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a22aa65f-fb55-49ec-a10e-c14a494422de-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"a22aa65f-fb55-49ec-a10e-c14a494422de\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 13 18:53:05.122685 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:05.122393    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a22aa65f-fb55-49ec-a10e-c14a494422de-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"a22aa65f-fb55-49ec-a10e-c14a494422de\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 13 18:53:05.123075 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:05.123055    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a22aa65f-fb55-49ec-a10e-c14a494422de-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"a22aa65f-fb55-49ec-a10e-c14a494422de\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 13 18:53:05.124315 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:05.124285    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a22aa65f-fb55-49ec-a10e-c14a494422de-tls-assets\") pod \"alertmanager-main-0\" (UID: \"a22aa65f-fb55-49ec-a10e-c14a494422de\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 13 18:53:05.124410 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:05.124392    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a22aa65f-fb55-49ec-a10e-c14a494422de-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"a22aa65f-fb55-49ec-a10e-c14a494422de\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 13 18:53:05.124529 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:05.124502    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a22aa65f-fb55-49ec-a10e-c14a494422de-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"a22aa65f-fb55-49ec-a10e-c14a494422de\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 13 18:53:05.124736 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:05.124690    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a22aa65f-fb55-49ec-a10e-c14a494422de-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"a22aa65f-fb55-49ec-a10e-c14a494422de\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 13 18:53:05.124736 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:05.124731    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a22aa65f-fb55-49ec-a10e-c14a494422de-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"a22aa65f-fb55-49ec-a10e-c14a494422de\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 13 18:53:05.124860 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:05.124816    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a22aa65f-fb55-49ec-a10e-c14a494422de-config-volume\") pod \"alertmanager-main-0\" (UID: \"a22aa65f-fb55-49ec-a10e-c14a494422de\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 13 18:53:05.125253 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:05.125236    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a22aa65f-fb55-49ec-a10e-c14a494422de-config-out\") pod \"alertmanager-main-0\" (UID: \"a22aa65f-fb55-49ec-a10e-c14a494422de\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 13 18:53:05.125915 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:05.125898    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a22aa65f-fb55-49ec-a10e-c14a494422de-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"a22aa65f-fb55-49ec-a10e-c14a494422de\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 13 18:53:05.126467 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:05.126449    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a22aa65f-fb55-49ec-a10e-c14a494422de-web-config\") pod \"alertmanager-main-0\" (UID: \"a22aa65f-fb55-49ec-a10e-c14a494422de\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 13 18:53:05.130956 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:05.130935    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvclf\" (UniqueName: \"kubernetes.io/projected/a22aa65f-fb55-49ec-a10e-c14a494422de-kube-api-access-nvclf\") pod \"alertmanager-main-0\" (UID: \"a22aa65f-fb55-49ec-a10e-c14a494422de\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 13 18:53:05.271514 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:05.271477    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 13 18:53:05.422015 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:05.421760    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 13 18:53:05.426200 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:53:05.426166    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda22aa65f_fb55_49ec_a10e_c14a494422de.slice/crio-af48f2d2e3ba0a733b582abee50270bd2309e024676d1b9a9695b81f260af478 WatchSource:0}: Error finding container af48f2d2e3ba0a733b582abee50270bd2309e024676d1b9a9695b81f260af478: Status 404 returned error can't find the container with id af48f2d2e3ba0a733b582abee50270bd2309e024676d1b9a9695b81f260af478
Jul 13 18:53:05.896268 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:05.896229    2565 generic.go:356] "Generic (PLEG): container finished" podID="a22aa65f-fb55-49ec-a10e-c14a494422de" containerID="bcadfe110ed2bd6bc66eef117b1a0260f9e98671cc435647075ebe2ecd96ae9c" exitCode=0
Jul 13 18:53:05.896718 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:05.896320    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a22aa65f-fb55-49ec-a10e-c14a494422de","Type":"ContainerDied","Data":"bcadfe110ed2bd6bc66eef117b1a0260f9e98671cc435647075ebe2ecd96ae9c"}
Jul 13 18:53:05.896718 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:05.896393    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a22aa65f-fb55-49ec-a10e-c14a494422de","Type":"ContainerStarted","Data":"af48f2d2e3ba0a733b582abee50270bd2309e024676d1b9a9695b81f260af478"}
Jul 13 18:53:06.054964 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:06.054932    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="882acd32-9687-4c39-9d84-4893191bccd3" path="/var/lib/kubelet/pods/882acd32-9687-4c39-9d84-4893191bccd3/volumes"
Jul 13 18:53:06.906964 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:06.906924    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a22aa65f-fb55-49ec-a10e-c14a494422de","Type":"ContainerStarted","Data":"50a501decb426b89462e40d0649430d1244717fe3d871b043b5f8ee0b5f82317"}
Jul 13 18:53:06.907317 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:06.907008    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a22aa65f-fb55-49ec-a10e-c14a494422de","Type":"ContainerStarted","Data":"cd5261e4d54dcc58a6aa804b1771f79119f310694ffe44541714643b4738cd82"}
Jul 13 18:53:06.907317 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:06.907049    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a22aa65f-fb55-49ec-a10e-c14a494422de","Type":"ContainerStarted","Data":"edb46340ce63ec7d4df9141dd7fa03b2f8b88bb733c65af9352a93823e9f306e"}
Jul 13 18:53:06.907317 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:06.907093    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a22aa65f-fb55-49ec-a10e-c14a494422de","Type":"ContainerStarted","Data":"29f1d10a5cce707c571c2c4bbeeec9e4abd1ac6030a50bcf951b30e9b1bbe81a"}
Jul 13 18:53:06.907317 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:06.907128    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a22aa65f-fb55-49ec-a10e-c14a494422de","Type":"ContainerStarted","Data":"47e00b817c081117506798559e3860a399df9864736078e2bdacb593c46cf7d9"}
Jul 13 18:53:06.907317 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:06.907168    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a22aa65f-fb55-49ec-a10e-c14a494422de","Type":"ContainerStarted","Data":"026190b212c9b2da006d7b5b0eaf4f5459691066f7e433f2d82ec6aeabbbcff1"}
Jul 13 18:53:06.938508 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:06.938448    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.922212471 podStartE2EDuration="2.938428203s" totalImagesPullingTime="0s" totalInitContainerRuntime="16.215732ms" isStatefulPod=false podCreationTimestamp="2026-07-13 18:53:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:53:06.935696223 +0000 UTC m=+173.460829659" watchObservedRunningTime="2026-07-13 18:53:06.938428203 +0000 UTC m=+173.463561650"
Jul 13 18:53:07.331755 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:07.331712    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-5699db478d-mjnwl"]
Jul 13 18:53:07.335756 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:07.335725    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5699db478d-mjnwl"
Jul 13 18:53:07.338831 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:07.338784    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Jul 13 18:53:07.338980 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:07.338866    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Jul 13 18:53:07.338980 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:07.338878    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Jul 13 18:53:07.338980 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:07.338926    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Jul 13 18:53:07.339171 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:07.338982    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-bhlrm\""
Jul 13 18:53:07.340329 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:07.340307    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Jul 13 18:53:07.346828 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:07.346759    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Jul 13 18:53:07.349327 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:07.349302    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5699db478d-mjnwl"]
Jul 13 18:53:07.443516 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:07.443483    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1c30602a-28cb-45df-9400-9e4d295cde13-metrics-client-ca\") pod \"telemeter-client-5699db478d-mjnwl\" (UID: \"1c30602a-28cb-45df-9400-9e4d295cde13\") " pod="openshift-monitoring/telemeter-client-5699db478d-mjnwl"
Jul 13 18:53:07.443747 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:07.443522    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/1c30602a-28cb-45df-9400-9e4d295cde13-telemeter-client-tls\") pod \"telemeter-client-5699db478d-mjnwl\" (UID: \"1c30602a-28cb-45df-9400-9e4d295cde13\") " pod="openshift-monitoring/telemeter-client-5699db478d-mjnwl"
Jul 13 18:53:07.443747 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:07.443553    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1c30602a-28cb-45df-9400-9e4d295cde13-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5699db478d-mjnwl\" (UID: \"1c30602a-28cb-45df-9400-9e4d295cde13\") " pod="openshift-monitoring/telemeter-client-5699db478d-mjnwl"
Jul 13 18:53:07.443747 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:07.443582    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/1c30602a-28cb-45df-9400-9e4d295cde13-secret-telemeter-client\") pod \"telemeter-client-5699db478d-mjnwl\" (UID: \"1c30602a-28cb-45df-9400-9e4d295cde13\") " pod="openshift-monitoring/telemeter-client-5699db478d-mjnwl"
Jul 13 18:53:07.443747 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:07.443607    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1c30602a-28cb-45df-9400-9e4d295cde13-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5699db478d-mjnwl\" (UID: \"1c30602a-28cb-45df-9400-9e4d295cde13\") " pod="openshift-monitoring/telemeter-client-5699db478d-mjnwl"
Jul 13 18:53:07.443747 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:07.443638    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/1c30602a-28cb-45df-9400-9e4d295cde13-federate-client-tls\") pod \"telemeter-client-5699db478d-mjnwl\" (UID: \"1c30602a-28cb-45df-9400-9e4d295cde13\") " pod="openshift-monitoring/telemeter-client-5699db478d-mjnwl"
Jul 13 18:53:07.443747 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:07.443663    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1c30602a-28cb-45df-9400-9e4d295cde13-serving-certs-ca-bundle\") pod \"telemeter-client-5699db478d-mjnwl\" (UID: \"1c30602a-28cb-45df-9400-9e4d295cde13\") " pod="openshift-monitoring/telemeter-client-5699db478d-mjnwl"
Jul 13 18:53:07.443747 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:07.443734    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47fcw\" (UniqueName: \"kubernetes.io/projected/1c30602a-28cb-45df-9400-9e4d295cde13-kube-api-access-47fcw\") pod \"telemeter-client-5699db478d-mjnwl\" (UID: \"1c30602a-28cb-45df-9400-9e4d295cde13\") " pod="openshift-monitoring/telemeter-client-5699db478d-mjnwl"
Jul 13 18:53:07.544445 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:07.544399    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/1c30602a-28cb-45df-9400-9e4d295cde13-secret-telemeter-client\") pod \"telemeter-client-5699db478d-mjnwl\" (UID: \"1c30602a-28cb-45df-9400-9e4d295cde13\") " pod="openshift-monitoring/telemeter-client-5699db478d-mjnwl"
Jul 13 18:53:07.544445 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:07.544444    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1c30602a-28cb-45df-9400-9e4d295cde13-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5699db478d-mjnwl\" (UID: \"1c30602a-28cb-45df-9400-9e4d295cde13\") " pod="openshift-monitoring/telemeter-client-5699db478d-mjnwl"
Jul 13 18:53:07.544754 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:07.544481    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/1c30602a-28cb-45df-9400-9e4d295cde13-federate-client-tls\") pod \"telemeter-client-5699db478d-mjnwl\" (UID: \"1c30602a-28cb-45df-9400-9e4d295cde13\") " pod="openshift-monitoring/telemeter-client-5699db478d-mjnwl"
Jul 13 18:53:07.544754 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:07.544512    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1c30602a-28cb-45df-9400-9e4d295cde13-serving-certs-ca-bundle\") pod \"telemeter-client-5699db478d-mjnwl\" (UID: \"1c30602a-28cb-45df-9400-9e4d295cde13\") " pod="openshift-monitoring/telemeter-client-5699db478d-mjnwl"
Jul 13 18:53:07.544754 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:07.544552    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47fcw\" (UniqueName: \"kubernetes.io/projected/1c30602a-28cb-45df-9400-9e4d295cde13-kube-api-access-47fcw\") pod \"telemeter-client-5699db478d-mjnwl\" (UID: \"1c30602a-28cb-45df-9400-9e4d295cde13\") " pod="openshift-monitoring/telemeter-client-5699db478d-mjnwl"
Jul 13 18:53:07.544754 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:07.544609    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1c30602a-28cb-45df-9400-9e4d295cde13-metrics-client-ca\") pod \"telemeter-client-5699db478d-mjnwl\" (UID: \"1c30602a-28cb-45df-9400-9e4d295cde13\") " pod="openshift-monitoring/telemeter-client-5699db478d-mjnwl"
Jul 13 18:53:07.544754 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:07.544636    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/1c30602a-28cb-45df-9400-9e4d295cde13-telemeter-client-tls\") pod \"telemeter-client-5699db478d-mjnwl\" (UID: \"1c30602a-28cb-45df-9400-9e4d295cde13\") " pod="openshift-monitoring/telemeter-client-5699db478d-mjnwl"
Jul 13 18:53:07.544754 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:07.544678    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1c30602a-28cb-45df-9400-9e4d295cde13-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5699db478d-mjnwl\" (UID: \"1c30602a-28cb-45df-9400-9e4d295cde13\") " pod="openshift-monitoring/telemeter-client-5699db478d-mjnwl"
Jul 13 18:53:07.545509 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:07.545446    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1c30602a-28cb-45df-9400-9e4d295cde13-serving-certs-ca-bundle\") pod \"telemeter-client-5699db478d-mjnwl\" (UID: \"1c30602a-28cb-45df-9400-9e4d295cde13\") " pod="openshift-monitoring/telemeter-client-5699db478d-mjnwl"
Jul 13 18:53:07.545854 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:07.545829    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1c30602a-28cb-45df-9400-9e4d295cde13-metrics-client-ca\") pod \"telemeter-client-5699db478d-mjnwl\" (UID: \"1c30602a-28cb-45df-9400-9e4d295cde13\") " pod="openshift-monitoring/telemeter-client-5699db478d-mjnwl"
Jul 13 18:53:07.545994 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:07.545971    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1c30602a-28cb-45df-9400-9e4d295cde13-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5699db478d-mjnwl\" (UID: \"1c30602a-28cb-45df-9400-9e4d295cde13\") " pod="openshift-monitoring/telemeter-client-5699db478d-mjnwl"
Jul 13 18:53:07.547597 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:07.547535    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/1c30602a-28cb-45df-9400-9e4d295cde13-secret-telemeter-client\") pod \"telemeter-client-5699db478d-mjnwl\" (UID: \"1c30602a-28cb-45df-9400-9e4d295cde13\") " pod="openshift-monitoring/telemeter-client-5699db478d-mjnwl"
Jul 13 18:53:07.547750 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:07.547619    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/1c30602a-28cb-45df-9400-9e4d295cde13-federate-client-tls\") pod \"telemeter-client-5699db478d-mjnwl\" (UID: \"1c30602a-28cb-45df-9400-9e4d295cde13\") " pod="openshift-monitoring/telemeter-client-5699db478d-mjnwl"
Jul 13 18:53:07.548164 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:07.548116    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/1c30602a-28cb-45df-9400-9e4d295cde13-telemeter-client-tls\") pod \"telemeter-client-5699db478d-mjnwl\" (UID: \"1c30602a-28cb-45df-9400-9e4d295cde13\") " pod="openshift-monitoring/telemeter-client-5699db478d-mjnwl"
Jul 13 18:53:07.548258 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:07.548180    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1c30602a-28cb-45df-9400-9e4d295cde13-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5699db478d-mjnwl\" (UID: \"1c30602a-28cb-45df-9400-9e4d295cde13\") " pod="openshift-monitoring/telemeter-client-5699db478d-mjnwl"
Jul 13 18:53:07.559339 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:07.559299    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-47fcw\" (UniqueName: \"kubernetes.io/projected/1c30602a-28cb-45df-9400-9e4d295cde13-kube-api-access-47fcw\") pod \"telemeter-client-5699db478d-mjnwl\" (UID: \"1c30602a-28cb-45df-9400-9e4d295cde13\") " pod="openshift-monitoring/telemeter-client-5699db478d-mjnwl"
Jul 13 18:53:07.647689 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:07.647601    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5699db478d-mjnwl"
Jul 13 18:53:07.817516 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:07.817480    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5699db478d-mjnwl"]
Jul 13 18:53:07.820598 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:53:07.820563    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c30602a_28cb_45df_9400_9e4d295cde13.slice/crio-bcb88033b37325b5f19e9a54c44d3774bf4add7acdeff8c208d2a6632bb38f19 WatchSource:0}: Error finding container bcb88033b37325b5f19e9a54c44d3774bf4add7acdeff8c208d2a6632bb38f19: Status 404 returned error can't find the container with id bcb88033b37325b5f19e9a54c44d3774bf4add7acdeff8c208d2a6632bb38f19
Jul 13 18:53:07.915847 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:07.915750    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5699db478d-mjnwl" event={"ID":"1c30602a-28cb-45df-9400-9e4d295cde13","Type":"ContainerStarted","Data":"bcb88033b37325b5f19e9a54c44d3774bf4add7acdeff8c208d2a6632bb38f19"}
Jul 13 18:53:09.926263 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:09.926224    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5699db478d-mjnwl" event={"ID":"1c30602a-28cb-45df-9400-9e4d295cde13","Type":"ContainerStarted","Data":"66dfd5c142660f04790a067c2de756ba836ad3fee01a0609b771622bd2d8d911"}
Jul 13 18:53:09.926697 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:09.926316    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5699db478d-mjnwl" event={"ID":"1c30602a-28cb-45df-9400-9e4d295cde13","Type":"ContainerStarted","Data":"c013d314179aee59d07ca247b4c7cd47be724d4f7a4d586cc6a263fd6c50c465"}
Jul 13 18:53:09.926697 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:09.926335    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5699db478d-mjnwl" event={"ID":"1c30602a-28cb-45df-9400-9e4d295cde13","Type":"ContainerStarted","Data":"a2531f70c38d63bf37a570753113e9d0c1bc795a24b7288c60c28f52bb1c0c4b"}
Jul 13 18:53:09.958167 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:09.958108    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-5699db478d-mjnwl" podStartSLOduration=1.342544408 podStartE2EDuration="2.958089128s" totalImagesPullingTime="1.61554472s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 18:53:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:53:09.9565496 +0000 UTC m=+176.481683042" watchObservedRunningTime="2026-07-13 18:53:09.958089128 +0000 UTC m=+176.483222543"
Jul 13 18:53:10.675307 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:10.675264    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-f6fd9dcd7-d8d6z"]
Jul 13 18:53:10.678810 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:10.678780    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-f6fd9dcd7-d8d6z"
Jul 13 18:53:10.701328 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:10.701294    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-f6fd9dcd7-d8d6z"]
Jul 13 18:53:10.774252 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:10.773997    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/63363952-85bf-44ba-a26b-0c19e7e3d31b-trusted-ca-bundle\") pod \"console-f6fd9dcd7-d8d6z\" (UID: \"63363952-85bf-44ba-a26b-0c19e7e3d31b\") " pod="openshift-console/console-f6fd9dcd7-d8d6z"
Jul 13 18:53:10.774252 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:10.774059    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/63363952-85bf-44ba-a26b-0c19e7e3d31b-service-ca\") pod \"console-f6fd9dcd7-d8d6z\" (UID: \"63363952-85bf-44ba-a26b-0c19e7e3d31b\") " pod="openshift-console/console-f6fd9dcd7-d8d6z"
Jul 13 18:53:10.774252 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:10.774102    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/63363952-85bf-44ba-a26b-0c19e7e3d31b-console-config\") pod \"console-f6fd9dcd7-d8d6z\" (UID: \"63363952-85bf-44ba-a26b-0c19e7e3d31b\") " pod="openshift-console/console-f6fd9dcd7-d8d6z"
Jul 13 18:53:10.774252 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:10.774124    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4kcr\" (UniqueName: \"kubernetes.io/projected/63363952-85bf-44ba-a26b-0c19e7e3d31b-kube-api-access-m4kcr\") pod \"console-f6fd9dcd7-d8d6z\" (UID: \"63363952-85bf-44ba-a26b-0c19e7e3d31b\") " pod="openshift-console/console-f6fd9dcd7-d8d6z"
Jul 13 18:53:10.774252 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:10.774160    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/63363952-85bf-44ba-a26b-0c19e7e3d31b-oauth-serving-cert\") pod \"console-f6fd9dcd7-d8d6z\" (UID: \"63363952-85bf-44ba-a26b-0c19e7e3d31b\") " pod="openshift-console/console-f6fd9dcd7-d8d6z"
Jul 13 18:53:10.774252 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:10.774229    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/63363952-85bf-44ba-a26b-0c19e7e3d31b-console-serving-cert\") pod \"console-f6fd9dcd7-d8d6z\" (UID: \"63363952-85bf-44ba-a26b-0c19e7e3d31b\") " pod="openshift-console/console-f6fd9dcd7-d8d6z"
Jul 13 18:53:10.774252 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:10.774265    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/63363952-85bf-44ba-a26b-0c19e7e3d31b-console-oauth-config\") pod \"console-f6fd9dcd7-d8d6z\" (UID: \"63363952-85bf-44ba-a26b-0c19e7e3d31b\") " pod="openshift-console/console-f6fd9dcd7-d8d6z"
Jul 13 18:53:10.875306 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:10.875258    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/63363952-85bf-44ba-a26b-0c19e7e3d31b-console-oauth-config\") pod \"console-f6fd9dcd7-d8d6z\" (UID: \"63363952-85bf-44ba-a26b-0c19e7e3d31b\") " pod="openshift-console/console-f6fd9dcd7-d8d6z"
Jul 13 18:53:10.876123 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:10.876096    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/63363952-85bf-44ba-a26b-0c19e7e3d31b-trusted-ca-bundle\") pod \"console-f6fd9dcd7-d8d6z\" (UID: \"63363952-85bf-44ba-a26b-0c19e7e3d31b\") " pod="openshift-console/console-f6fd9dcd7-d8d6z"
Jul 13 18:53:10.876488 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:10.876459    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/63363952-85bf-44ba-a26b-0c19e7e3d31b-service-ca\") pod \"console-f6fd9dcd7-d8d6z\" (UID: \"63363952-85bf-44ba-a26b-0c19e7e3d31b\") " pod="openshift-console/console-f6fd9dcd7-d8d6z"
Jul 13 18:53:10.876613 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:10.876528    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/63363952-85bf-44ba-a26b-0c19e7e3d31b-console-config\") pod \"console-f6fd9dcd7-d8d6z\" (UID: \"63363952-85bf-44ba-a26b-0c19e7e3d31b\") " pod="openshift-console/console-f6fd9dcd7-d8d6z"
Jul 13 18:53:10.876613 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:10.876553    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4kcr\" (UniqueName: \"kubernetes.io/projected/63363952-85bf-44ba-a26b-0c19e7e3d31b-kube-api-access-m4kcr\") pod \"console-f6fd9dcd7-d8d6z\" (UID: \"63363952-85bf-44ba-a26b-0c19e7e3d31b\") " pod="openshift-console/console-f6fd9dcd7-d8d6z"
Jul 13 18:53:10.876613 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:10.876590    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/63363952-85bf-44ba-a26b-0c19e7e3d31b-oauth-serving-cert\") pod \"console-f6fd9dcd7-d8d6z\" (UID: \"63363952-85bf-44ba-a26b-0c19e7e3d31b\") " pod="openshift-console/console-f6fd9dcd7-d8d6z"
Jul 13 18:53:10.876808 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:10.876683    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/63363952-85bf-44ba-a26b-0c19e7e3d31b-console-serving-cert\") pod \"console-f6fd9dcd7-d8d6z\" (UID: \"63363952-85bf-44ba-a26b-0c19e7e3d31b\") " pod="openshift-console/console-f6fd9dcd7-d8d6z"
Jul 13 18:53:10.877190 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:10.877164    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/63363952-85bf-44ba-a26b-0c19e7e3d31b-trusted-ca-bundle\") pod \"console-f6fd9dcd7-d8d6z\" (UID: \"63363952-85bf-44ba-a26b-0c19e7e3d31b\") " pod="openshift-console/console-f6fd9dcd7-d8d6z"
Jul 13 18:53:10.877506 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:10.877482    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/63363952-85bf-44ba-a26b-0c19e7e3d31b-oauth-serving-cert\") pod \"console-f6fd9dcd7-d8d6z\" (UID: \"63363952-85bf-44ba-a26b-0c19e7e3d31b\") " pod="openshift-console/console-f6fd9dcd7-d8d6z"
Jul 13 18:53:10.877614 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:10.877589    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/63363952-85bf-44ba-a26b-0c19e7e3d31b-console-config\") pod \"console-f6fd9dcd7-d8d6z\" (UID: \"63363952-85bf-44ba-a26b-0c19e7e3d31b\") " pod="openshift-console/console-f6fd9dcd7-d8d6z"
Jul 13 18:53:10.877757 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:10.877733    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/63363952-85bf-44ba-a26b-0c19e7e3d31b-service-ca\") pod \"console-f6fd9dcd7-d8d6z\" (UID: \"63363952-85bf-44ba-a26b-0c19e7e3d31b\") " pod="openshift-console/console-f6fd9dcd7-d8d6z"
Jul 13 18:53:10.879011 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:10.878989    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/63363952-85bf-44ba-a26b-0c19e7e3d31b-console-oauth-config\") pod \"console-f6fd9dcd7-d8d6z\" (UID: \"63363952-85bf-44ba-a26b-0c19e7e3d31b\") " pod="openshift-console/console-f6fd9dcd7-d8d6z"
Jul 13 18:53:10.879946 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:10.879920    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/63363952-85bf-44ba-a26b-0c19e7e3d31b-console-serving-cert\") pod \"console-f6fd9dcd7-d8d6z\" (UID: \"63363952-85bf-44ba-a26b-0c19e7e3d31b\") " pod="openshift-console/console-f6fd9dcd7-d8d6z"
Jul 13 18:53:10.887988 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:10.887949    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4kcr\" (UniqueName: \"kubernetes.io/projected/63363952-85bf-44ba-a26b-0c19e7e3d31b-kube-api-access-m4kcr\") pod \"console-f6fd9dcd7-d8d6z\" (UID: \"63363952-85bf-44ba-a26b-0c19e7e3d31b\") " pod="openshift-console/console-f6fd9dcd7-d8d6z"
Jul 13 18:53:10.988671 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:10.988560    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-f6fd9dcd7-d8d6z"
Jul 13 18:53:11.132222 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:11.132185    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-f6fd9dcd7-d8d6z"]
Jul 13 18:53:11.136167 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:53:11.136137    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63363952_85bf_44ba_a26b_0c19e7e3d31b.slice/crio-68e533b582c3058cefed70710d4c48480f213fe8faeafa19165e0432ee32efb5 WatchSource:0}: Error finding container 68e533b582c3058cefed70710d4c48480f213fe8faeafa19165e0432ee32efb5: Status 404 returned error can't find the container with id 68e533b582c3058cefed70710d4c48480f213fe8faeafa19165e0432ee32efb5
Jul 13 18:53:11.937515 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:11.937474    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-f6fd9dcd7-d8d6z" event={"ID":"63363952-85bf-44ba-a26b-0c19e7e3d31b","Type":"ContainerStarted","Data":"125505b8d32364873b224964649914fb8fd24a71fffbff142bb374965f0a0253"}
Jul 13 18:53:11.937695 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:11.937537    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-f6fd9dcd7-d8d6z" event={"ID":"63363952-85bf-44ba-a26b-0c19e7e3d31b","Type":"ContainerStarted","Data":"68e533b582c3058cefed70710d4c48480f213fe8faeafa19165e0432ee32efb5"}
Jul 13 18:53:11.956816 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:11.956759    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-f6fd9dcd7-d8d6z" podStartSLOduration=1.956739703 podStartE2EDuration="1.956739703s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 18:53:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:53:11.955544456 +0000 UTC m=+178.480677872" watchObservedRunningTime="2026-07-13 18:53:11.956739703 +0000 UTC m=+178.481873118"
Jul 13 18:53:20.989067 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:20.989025    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-f6fd9dcd7-d8d6z"
Jul 13 18:53:20.989569 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:20.989104    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-f6fd9dcd7-d8d6z"
Jul 13 18:53:20.995173 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:20.995147    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-f6fd9dcd7-d8d6z"
Jul 13 18:53:21.984031 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:21.983999    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-f6fd9dcd7-d8d6z"
Jul 13 18:53:22.036993 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:22.036959    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-f45bd79cb-2gcbm"]
Jul 13 18:53:47.065576 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:47.065510    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-f45bd79cb-2gcbm" podUID="cd892f91-5591-49e8-bdfa-c455803db4fc" containerName="console" containerID="cri-o://8c2a54176fe0d846aacec4f7c2b8521b1712c031cc19f2fa0f554e18776e9b03" gracePeriod=15
Jul 13 18:53:47.316522 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:47.313746    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-f45bd79cb-2gcbm"
Jul 13 18:53:47.392392 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:47.392345    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cd892f91-5591-49e8-bdfa-c455803db4fc-service-ca\") pod \"cd892f91-5591-49e8-bdfa-c455803db4fc\" (UID: \"cd892f91-5591-49e8-bdfa-c455803db4fc\") "
Jul 13 18:53:47.392593 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:47.392417    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wcjl2\" (UniqueName: \"kubernetes.io/projected/cd892f91-5591-49e8-bdfa-c455803db4fc-kube-api-access-wcjl2\") pod \"cd892f91-5591-49e8-bdfa-c455803db4fc\" (UID: \"cd892f91-5591-49e8-bdfa-c455803db4fc\") "
Jul 13 18:53:47.392593 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:47.392471    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cd892f91-5591-49e8-bdfa-c455803db4fc-console-serving-cert\") pod \"cd892f91-5591-49e8-bdfa-c455803db4fc\" (UID: \"cd892f91-5591-49e8-bdfa-c455803db4fc\") "
Jul 13 18:53:47.392593 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:47.392507    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cd892f91-5591-49e8-bdfa-c455803db4fc-console-oauth-config\") pod \"cd892f91-5591-49e8-bdfa-c455803db4fc\" (UID: \"cd892f91-5591-49e8-bdfa-c455803db4fc\") "
Jul 13 18:53:47.392593 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:47.392548    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cd892f91-5591-49e8-bdfa-c455803db4fc-trusted-ca-bundle\") pod \"cd892f91-5591-49e8-bdfa-c455803db4fc\" (UID: \"cd892f91-5591-49e8-bdfa-c455803db4fc\") "
Jul 13 18:53:47.392593 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:47.392569    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cd892f91-5591-49e8-bdfa-c455803db4fc-oauth-serving-cert\") pod \"cd892f91-5591-49e8-bdfa-c455803db4fc\" (UID: \"cd892f91-5591-49e8-bdfa-c455803db4fc\") "
Jul 13 18:53:47.392874 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:47.392601    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cd892f91-5591-49e8-bdfa-c455803db4fc-console-config\") pod \"cd892f91-5591-49e8-bdfa-c455803db4fc\" (UID: \"cd892f91-5591-49e8-bdfa-c455803db4fc\") "
Jul 13 18:53:47.392874 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:47.392850    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd892f91-5591-49e8-bdfa-c455803db4fc-service-ca" (OuterVolumeSpecName: "service-ca") pod "cd892f91-5591-49e8-bdfa-c455803db4fc" (UID: "cd892f91-5591-49e8-bdfa-c455803db4fc"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 18:53:47.393077 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:47.393045    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd892f91-5591-49e8-bdfa-c455803db4fc-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "cd892f91-5591-49e8-bdfa-c455803db4fc" (UID: "cd892f91-5591-49e8-bdfa-c455803db4fc"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 18:53:47.393204 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:47.393166    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd892f91-5591-49e8-bdfa-c455803db4fc-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "cd892f91-5591-49e8-bdfa-c455803db4fc" (UID: "cd892f91-5591-49e8-bdfa-c455803db4fc"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 18:53:47.393204 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:47.393028    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd892f91-5591-49e8-bdfa-c455803db4fc-console-config" (OuterVolumeSpecName: "console-config") pod "cd892f91-5591-49e8-bdfa-c455803db4fc" (UID: "cd892f91-5591-49e8-bdfa-c455803db4fc"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 18:53:47.395008 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:47.394976    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd892f91-5591-49e8-bdfa-c455803db4fc-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "cd892f91-5591-49e8-bdfa-c455803db4fc" (UID: "cd892f91-5591-49e8-bdfa-c455803db4fc"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 18:53:47.395008 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:47.394999    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd892f91-5591-49e8-bdfa-c455803db4fc-kube-api-access-wcjl2" (OuterVolumeSpecName: "kube-api-access-wcjl2") pod "cd892f91-5591-49e8-bdfa-c455803db4fc" (UID: "cd892f91-5591-49e8-bdfa-c455803db4fc"). InnerVolumeSpecName "kube-api-access-wcjl2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 18:53:47.395165 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:47.395094    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd892f91-5591-49e8-bdfa-c455803db4fc-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "cd892f91-5591-49e8-bdfa-c455803db4fc" (UID: "cd892f91-5591-49e8-bdfa-c455803db4fc"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 18:53:47.493947 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:47.493908    2565 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cd892f91-5591-49e8-bdfa-c455803db4fc-console-serving-cert\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 13 18:53:47.493947 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:47.493943    2565 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cd892f91-5591-49e8-bdfa-c455803db4fc-console-oauth-config\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 13 18:53:47.493947 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:47.493954    2565 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cd892f91-5591-49e8-bdfa-c455803db4fc-trusted-ca-bundle\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 13 18:53:47.494183 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:47.493963    2565 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cd892f91-5591-49e8-bdfa-c455803db4fc-oauth-serving-cert\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 13 18:53:47.494183 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:47.493972    2565 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cd892f91-5591-49e8-bdfa-c455803db4fc-console-config\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 13 18:53:47.494183 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:47.493981    2565 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cd892f91-5591-49e8-bdfa-c455803db4fc-service-ca\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 13 18:53:47.494183 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:47.493988    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wcjl2\" (UniqueName: \"kubernetes.io/projected/cd892f91-5591-49e8-bdfa-c455803db4fc-kube-api-access-wcjl2\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 13 18:53:48.092145 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:48.092112    2565 generic.go:356] "Generic (PLEG): container finished" podID="cd892f91-5591-49e8-bdfa-c455803db4fc" containerID="8c2a54176fe0d846aacec4f7c2b8521b1712c031cc19f2fa0f554e18776e9b03" exitCode=0
Jul 13 18:53:48.092615 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:48.092162    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-f45bd79cb-2gcbm"
Jul 13 18:53:48.092615 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:48.092197    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-f45bd79cb-2gcbm" event={"ID":"cd892f91-5591-49e8-bdfa-c455803db4fc","Type":"ContainerDied","Data":"8c2a54176fe0d846aacec4f7c2b8521b1712c031cc19f2fa0f554e18776e9b03"}
Jul 13 18:53:48.092615 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:48.092264    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-f45bd79cb-2gcbm" event={"ID":"cd892f91-5591-49e8-bdfa-c455803db4fc","Type":"ContainerDied","Data":"d7d89f49a7b01b16993045a94d97ce7ebaede4ef595e2b696727371b4c4d8940"}
Jul 13 18:53:48.092615 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:48.092296    2565 scope.go:117] "RemoveContainer" containerID="8c2a54176fe0d846aacec4f7c2b8521b1712c031cc19f2fa0f554e18776e9b03"
Jul 13 18:53:48.103579 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:48.103553    2565 scope.go:117] "RemoveContainer" containerID="8c2a54176fe0d846aacec4f7c2b8521b1712c031cc19f2fa0f554e18776e9b03"
Jul 13 18:53:48.103932 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:53:48.103911    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c2a54176fe0d846aacec4f7c2b8521b1712c031cc19f2fa0f554e18776e9b03\": container with ID starting with 8c2a54176fe0d846aacec4f7c2b8521b1712c031cc19f2fa0f554e18776e9b03 not found: ID does not exist" containerID="8c2a54176fe0d846aacec4f7c2b8521b1712c031cc19f2fa0f554e18776e9b03"
Jul 13 18:53:48.104013 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:48.103942    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c2a54176fe0d846aacec4f7c2b8521b1712c031cc19f2fa0f554e18776e9b03"} err="failed to get container status \"8c2a54176fe0d846aacec4f7c2b8521b1712c031cc19f2fa0f554e18776e9b03\": rpc error: code = NotFound desc = could not find container \"8c2a54176fe0d846aacec4f7c2b8521b1712c031cc19f2fa0f554e18776e9b03\": container with ID starting with 8c2a54176fe0d846aacec4f7c2b8521b1712c031cc19f2fa0f554e18776e9b03 not found: ID does not exist"
Jul 13 18:53:48.109847 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:48.109818    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-f45bd79cb-2gcbm"]
Jul 13 18:53:48.111202 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:48.111182    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-f45bd79cb-2gcbm"]
Jul 13 18:53:50.047059 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:53:50.047029    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cd892f91-5591-49e8-bdfa-c455803db4fc" path="/var/lib/kubelet/pods/cd892f91-5591-49e8-bdfa-c455803db4fc/volumes"
Jul 13 18:54:20.609166 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:54:20.609123    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-95b84c78f-ggrgb"]
Jul 13 18:54:20.609787 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:54:20.609621    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd892f91-5591-49e8-bdfa-c455803db4fc" containerName="console"
Jul 13 18:54:20.609787 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:54:20.609641    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd892f91-5591-49e8-bdfa-c455803db4fc" containerName="console"
Jul 13 18:54:20.609787 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:54:20.609773    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="cd892f91-5591-49e8-bdfa-c455803db4fc" containerName="console"
Jul 13 18:54:20.614138 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:54:20.614113    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-95b84c78f-ggrgb"
Jul 13 18:54:20.624535 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:54:20.624506    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-95b84c78f-ggrgb"]
Jul 13 18:54:20.803692 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:54:20.803650    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/030384be-1be3-4bd0-8ba2-7cf6df3ba223-console-oauth-config\") pod \"console-95b84c78f-ggrgb\" (UID: \"030384be-1be3-4bd0-8ba2-7cf6df3ba223\") " pod="openshift-console/console-95b84c78f-ggrgb"
Jul 13 18:54:20.803692 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:54:20.803690    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/030384be-1be3-4bd0-8ba2-7cf6df3ba223-console-config\") pod \"console-95b84c78f-ggrgb\" (UID: \"030384be-1be3-4bd0-8ba2-7cf6df3ba223\") " pod="openshift-console/console-95b84c78f-ggrgb"
Jul 13 18:54:20.803939 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:54:20.803829    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/030384be-1be3-4bd0-8ba2-7cf6df3ba223-service-ca\") pod \"console-95b84c78f-ggrgb\" (UID: \"030384be-1be3-4bd0-8ba2-7cf6df3ba223\") " pod="openshift-console/console-95b84c78f-ggrgb"
Jul 13 18:54:20.803939 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:54:20.803867    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/030384be-1be3-4bd0-8ba2-7cf6df3ba223-console-serving-cert\") pod \"console-95b84c78f-ggrgb\" (UID: \"030384be-1be3-4bd0-8ba2-7cf6df3ba223\") " pod="openshift-console/console-95b84c78f-ggrgb"
Jul 13 18:54:20.803939 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:54:20.803906    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/030384be-1be3-4bd0-8ba2-7cf6df3ba223-trusted-ca-bundle\") pod \"console-95b84c78f-ggrgb\" (UID: \"030384be-1be3-4bd0-8ba2-7cf6df3ba223\") " pod="openshift-console/console-95b84c78f-ggrgb"
Jul 13 18:54:20.804031 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:54:20.803956    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/030384be-1be3-4bd0-8ba2-7cf6df3ba223-oauth-serving-cert\") pod \"console-95b84c78f-ggrgb\" (UID: \"030384be-1be3-4bd0-8ba2-7cf6df3ba223\") " pod="openshift-console/console-95b84c78f-ggrgb"
Jul 13 18:54:20.804031 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:54:20.804008    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkxfx\" (UniqueName: \"kubernetes.io/projected/030384be-1be3-4bd0-8ba2-7cf6df3ba223-kube-api-access-jkxfx\") pod \"console-95b84c78f-ggrgb\" (UID: \"030384be-1be3-4bd0-8ba2-7cf6df3ba223\") " pod="openshift-console/console-95b84c78f-ggrgb"
Jul 13 18:54:20.905185 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:54:20.905085    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/030384be-1be3-4bd0-8ba2-7cf6df3ba223-trusted-ca-bundle\") pod \"console-95b84c78f-ggrgb\" (UID: \"030384be-1be3-4bd0-8ba2-7cf6df3ba223\") " pod="openshift-console/console-95b84c78f-ggrgb"
Jul 13 18:54:20.905185 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:54:20.905131    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/030384be-1be3-4bd0-8ba2-7cf6df3ba223-oauth-serving-cert\") pod \"console-95b84c78f-ggrgb\" (UID: \"030384be-1be3-4bd0-8ba2-7cf6df3ba223\") " pod="openshift-console/console-95b84c78f-ggrgb"
Jul 13 18:54:20.906265 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:54:20.905260    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkxfx\" (UniqueName: \"kubernetes.io/projected/030384be-1be3-4bd0-8ba2-7cf6df3ba223-kube-api-access-jkxfx\") pod \"console-95b84c78f-ggrgb\" (UID: \"030384be-1be3-4bd0-8ba2-7cf6df3ba223\") " pod="openshift-console/console-95b84c78f-ggrgb"
Jul 13 18:54:20.906265 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:54:20.905731    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/030384be-1be3-4bd0-8ba2-7cf6df3ba223-console-oauth-config\") pod \"console-95b84c78f-ggrgb\" (UID: \"030384be-1be3-4bd0-8ba2-7cf6df3ba223\") " pod="openshift-console/console-95b84c78f-ggrgb"
Jul 13 18:54:20.906265 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:54:20.905788    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/030384be-1be3-4bd0-8ba2-7cf6df3ba223-console-config\") pod \"console-95b84c78f-ggrgb\" (UID: \"030384be-1be3-4bd0-8ba2-7cf6df3ba223\") " pod="openshift-console/console-95b84c78f-ggrgb"
Jul 13 18:54:20.906265 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:54:20.906212    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/030384be-1be3-4bd0-8ba2-7cf6df3ba223-oauth-serving-cert\") pod \"console-95b84c78f-ggrgb\" (UID: \"030384be-1be3-4bd0-8ba2-7cf6df3ba223\") " pod="openshift-console/console-95b84c78f-ggrgb"
Jul 13 18:54:20.906551 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:54:20.906529    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/030384be-1be3-4bd0-8ba2-7cf6df3ba223-trusted-ca-bundle\") pod \"console-95b84c78f-ggrgb\" (UID: \"030384be-1be3-4bd0-8ba2-7cf6df3ba223\") " pod="openshift-console/console-95b84c78f-ggrgb"
Jul 13 18:54:20.906610 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:54:20.906544    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/030384be-1be3-4bd0-8ba2-7cf6df3ba223-console-config\") pod \"console-95b84c78f-ggrgb\" (UID: \"030384be-1be3-4bd0-8ba2-7cf6df3ba223\") " pod="openshift-console/console-95b84c78f-ggrgb"
Jul 13 18:54:20.906663 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:54:20.905875    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/030384be-1be3-4bd0-8ba2-7cf6df3ba223-service-ca\") pod \"console-95b84c78f-ggrgb\" (UID: \"030384be-1be3-4bd0-8ba2-7cf6df3ba223\") " pod="openshift-console/console-95b84c78f-ggrgb"
Jul 13 18:54:20.912404 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:54:20.906876    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/030384be-1be3-4bd0-8ba2-7cf6df3ba223-service-ca\") pod \"console-95b84c78f-ggrgb\" (UID: \"030384be-1be3-4bd0-8ba2-7cf6df3ba223\") " pod="openshift-console/console-95b84c78f-ggrgb"
Jul 13 18:54:20.912404 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:54:20.906881    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/030384be-1be3-4bd0-8ba2-7cf6df3ba223-console-serving-cert\") pod \"console-95b84c78f-ggrgb\" (UID: \"030384be-1be3-4bd0-8ba2-7cf6df3ba223\") " pod="openshift-console/console-95b84c78f-ggrgb"
Jul 13 18:54:20.913306 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:54:20.913283    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/030384be-1be3-4bd0-8ba2-7cf6df3ba223-console-oauth-config\") pod \"console-95b84c78f-ggrgb\" (UID: \"030384be-1be3-4bd0-8ba2-7cf6df3ba223\") " pod="openshift-console/console-95b84c78f-ggrgb"
Jul 13 18:54:20.913362 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:54:20.913314    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/030384be-1be3-4bd0-8ba2-7cf6df3ba223-console-serving-cert\") pod \"console-95b84c78f-ggrgb\" (UID: \"030384be-1be3-4bd0-8ba2-7cf6df3ba223\") " pod="openshift-console/console-95b84c78f-ggrgb"
Jul 13 18:54:20.915558 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:54:20.915527    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkxfx\" (UniqueName: \"kubernetes.io/projected/030384be-1be3-4bd0-8ba2-7cf6df3ba223-kube-api-access-jkxfx\") pod \"console-95b84c78f-ggrgb\" (UID: \"030384be-1be3-4bd0-8ba2-7cf6df3ba223\") " pod="openshift-console/console-95b84c78f-ggrgb"
Jul 13 18:54:20.924465 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:54:20.924434    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-95b84c78f-ggrgb"
Jul 13 18:54:21.064923 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:54:21.064899    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-95b84c78f-ggrgb"]
Jul 13 18:54:21.067761 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:54:21.067732    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod030384be_1be3_4bd0_8ba2_7cf6df3ba223.slice/crio-7b5f089b396a27edcadbc0a921db23be0ce01ffac174fadc593faa938e97e49b WatchSource:0}: Error finding container 7b5f089b396a27edcadbc0a921db23be0ce01ffac174fadc593faa938e97e49b: Status 404 returned error can't find the container with id 7b5f089b396a27edcadbc0a921db23be0ce01ffac174fadc593faa938e97e49b
Jul 13 18:54:21.231117 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:54:21.231030    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-95b84c78f-ggrgb" event={"ID":"030384be-1be3-4bd0-8ba2-7cf6df3ba223","Type":"ContainerStarted","Data":"091f84aa260dcbc8789700c057e13e8ed14b00e1e8f3074f2329ba2407265a9f"}
Jul 13 18:54:21.231117 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:54:21.231098    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-95b84c78f-ggrgb" event={"ID":"030384be-1be3-4bd0-8ba2-7cf6df3ba223","Type":"ContainerStarted","Data":"7b5f089b396a27edcadbc0a921db23be0ce01ffac174fadc593faa938e97e49b"}
Jul 13 18:54:21.251958 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:54:21.251900    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-95b84c78f-ggrgb" podStartSLOduration=1.251881467 podStartE2EDuration="1.251881467s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 18:54:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:54:21.24974483 +0000 UTC m=+247.774878265" watchObservedRunningTime="2026-07-13 18:54:21.251881467 +0000 UTC m=+247.777014956"
Jul 13 18:54:30.925459 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:54:30.925417    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-95b84c78f-ggrgb"
Jul 13 18:54:30.926042 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:54:30.925492    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-95b84c78f-ggrgb"
Jul 13 18:54:30.930175 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:54:30.930149    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-95b84c78f-ggrgb"
Jul 13 18:54:31.278672 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:54:31.278642    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-95b84c78f-ggrgb"
Jul 13 18:54:31.332384 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:54:31.332343    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-f6fd9dcd7-d8d6z"]
Jul 13 18:54:56.354363 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:54:56.354296    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-f6fd9dcd7-d8d6z" podUID="63363952-85bf-44ba-a26b-0c19e7e3d31b" containerName="console" containerID="cri-o://125505b8d32364873b224964649914fb8fd24a71fffbff142bb374965f0a0253" gracePeriod=15
Jul 13 18:54:56.610225 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:54:56.610146    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-f6fd9dcd7-d8d6z"
Jul 13 18:54:56.729513 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:54:56.729469    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/63363952-85bf-44ba-a26b-0c19e7e3d31b-service-ca\") pod \"63363952-85bf-44ba-a26b-0c19e7e3d31b\" (UID: \"63363952-85bf-44ba-a26b-0c19e7e3d31b\") "
Jul 13 18:54:56.729696 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:54:56.729521    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/63363952-85bf-44ba-a26b-0c19e7e3d31b-console-serving-cert\") pod \"63363952-85bf-44ba-a26b-0c19e7e3d31b\" (UID: \"63363952-85bf-44ba-a26b-0c19e7e3d31b\") "
Jul 13 18:54:56.729696 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:54:56.729571    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/63363952-85bf-44ba-a26b-0c19e7e3d31b-console-oauth-config\") pod \"63363952-85bf-44ba-a26b-0c19e7e3d31b\" (UID: \"63363952-85bf-44ba-a26b-0c19e7e3d31b\") "
Jul 13 18:54:56.729696 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:54:56.729643    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/63363952-85bf-44ba-a26b-0c19e7e3d31b-oauth-serving-cert\") pod \"63363952-85bf-44ba-a26b-0c19e7e3d31b\" (UID: \"63363952-85bf-44ba-a26b-0c19e7e3d31b\") "
Jul 13 18:54:56.729696 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:54:56.729692    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/63363952-85bf-44ba-a26b-0c19e7e3d31b-trusted-ca-bundle\") pod \"63363952-85bf-44ba-a26b-0c19e7e3d31b\" (UID: \"63363952-85bf-44ba-a26b-0c19e7e3d31b\") "
Jul 13 18:54:56.730008 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:54:56.729750    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/63363952-85bf-44ba-a26b-0c19e7e3d31b-console-config\") pod \"63363952-85bf-44ba-a26b-0c19e7e3d31b\" (UID: \"63363952-85bf-44ba-a26b-0c19e7e3d31b\") "
Jul 13 18:54:56.730008 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:54:56.729778    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m4kcr\" (UniqueName: \"kubernetes.io/projected/63363952-85bf-44ba-a26b-0c19e7e3d31b-kube-api-access-m4kcr\") pod \"63363952-85bf-44ba-a26b-0c19e7e3d31b\" (UID: \"63363952-85bf-44ba-a26b-0c19e7e3d31b\") "
Jul 13 18:54:56.730114 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:54:56.730054    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63363952-85bf-44ba-a26b-0c19e7e3d31b-service-ca" (OuterVolumeSpecName: "service-ca") pod "63363952-85bf-44ba-a26b-0c19e7e3d31b" (UID: "63363952-85bf-44ba-a26b-0c19e7e3d31b"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 18:54:56.730167 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:54:56.730122    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63363952-85bf-44ba-a26b-0c19e7e3d31b-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "63363952-85bf-44ba-a26b-0c19e7e3d31b" (UID: "63363952-85bf-44ba-a26b-0c19e7e3d31b"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 18:54:56.730206 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:54:56.730169    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63363952-85bf-44ba-a26b-0c19e7e3d31b-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "63363952-85bf-44ba-a26b-0c19e7e3d31b" (UID: "63363952-85bf-44ba-a26b-0c19e7e3d31b"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 18:54:56.730318 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:54:56.730289    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63363952-85bf-44ba-a26b-0c19e7e3d31b-console-config" (OuterVolumeSpecName: "console-config") pod "63363952-85bf-44ba-a26b-0c19e7e3d31b" (UID: "63363952-85bf-44ba-a26b-0c19e7e3d31b"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 18:54:56.731980 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:54:56.731949    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63363952-85bf-44ba-a26b-0c19e7e3d31b-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "63363952-85bf-44ba-a26b-0c19e7e3d31b" (UID: "63363952-85bf-44ba-a26b-0c19e7e3d31b"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 18:54:56.732100 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:54:56.732005    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63363952-85bf-44ba-a26b-0c19e7e3d31b-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "63363952-85bf-44ba-a26b-0c19e7e3d31b" (UID: "63363952-85bf-44ba-a26b-0c19e7e3d31b"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 18:54:56.732162 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:54:56.732106    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63363952-85bf-44ba-a26b-0c19e7e3d31b-kube-api-access-m4kcr" (OuterVolumeSpecName: "kube-api-access-m4kcr") pod "63363952-85bf-44ba-a26b-0c19e7e3d31b" (UID: "63363952-85bf-44ba-a26b-0c19e7e3d31b"). InnerVolumeSpecName "kube-api-access-m4kcr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 18:54:56.831228 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:54:56.831179    2565 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/63363952-85bf-44ba-a26b-0c19e7e3d31b-trusted-ca-bundle\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 13 18:54:56.831228 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:54:56.831217    2565 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/63363952-85bf-44ba-a26b-0c19e7e3d31b-console-config\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 13 18:54:56.831228 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:54:56.831231    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m4kcr\" (UniqueName: \"kubernetes.io/projected/63363952-85bf-44ba-a26b-0c19e7e3d31b-kube-api-access-m4kcr\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 13 18:54:56.831479 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:54:56.831245    2565 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/63363952-85bf-44ba-a26b-0c19e7e3d31b-service-ca\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 13 18:54:56.831479 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:54:56.831257    2565 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/63363952-85bf-44ba-a26b-0c19e7e3d31b-console-serving-cert\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 13 18:54:56.831479 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:54:56.831268    2565 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/63363952-85bf-44ba-a26b-0c19e7e3d31b-console-oauth-config\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 13 18:54:56.831479 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:54:56.831279    2565 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/63363952-85bf-44ba-a26b-0c19e7e3d31b-oauth-serving-cert\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 13 18:54:57.391867 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:54:57.391828    2565 generic.go:356] "Generic (PLEG): container finished" podID="63363952-85bf-44ba-a26b-0c19e7e3d31b" containerID="125505b8d32364873b224964649914fb8fd24a71fffbff142bb374965f0a0253" exitCode=0
Jul 13 18:54:57.392355 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:54:57.391883    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-f6fd9dcd7-d8d6z"
Jul 13 18:54:57.392355 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:54:57.391887    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-f6fd9dcd7-d8d6z" event={"ID":"63363952-85bf-44ba-a26b-0c19e7e3d31b","Type":"ContainerDied","Data":"125505b8d32364873b224964649914fb8fd24a71fffbff142bb374965f0a0253"}
Jul 13 18:54:57.392355 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:54:57.392001    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-f6fd9dcd7-d8d6z" event={"ID":"63363952-85bf-44ba-a26b-0c19e7e3d31b","Type":"ContainerDied","Data":"68e533b582c3058cefed70710d4c48480f213fe8faeafa19165e0432ee32efb5"}
Jul 13 18:54:57.392355 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:54:57.392023    2565 scope.go:117] "RemoveContainer" containerID="125505b8d32364873b224964649914fb8fd24a71fffbff142bb374965f0a0253"
Jul 13 18:54:57.400775 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:54:57.400755    2565 scope.go:117] "RemoveContainer" containerID="125505b8d32364873b224964649914fb8fd24a71fffbff142bb374965f0a0253"
Jul 13 18:54:57.401072 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:54:57.401054    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"125505b8d32364873b224964649914fb8fd24a71fffbff142bb374965f0a0253\": container with ID starting with 125505b8d32364873b224964649914fb8fd24a71fffbff142bb374965f0a0253 not found: ID does not exist" containerID="125505b8d32364873b224964649914fb8fd24a71fffbff142bb374965f0a0253"
Jul 13 18:54:57.401135 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:54:57.401082    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"125505b8d32364873b224964649914fb8fd24a71fffbff142bb374965f0a0253"} err="failed to get container status \"125505b8d32364873b224964649914fb8fd24a71fffbff142bb374965f0a0253\": rpc error: code = NotFound desc = could not find container \"125505b8d32364873b224964649914fb8fd24a71fffbff142bb374965f0a0253\": container with ID starting with 125505b8d32364873b224964649914fb8fd24a71fffbff142bb374965f0a0253 not found: ID does not exist"
Jul 13 18:54:57.416034 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:54:57.416000    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-f6fd9dcd7-d8d6z"]
Jul 13 18:54:57.420254 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:54:57.420231    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-f6fd9dcd7-d8d6z"]
Jul 13 18:54:58.048033 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:54:58.048004    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="63363952-85bf-44ba-a26b-0c19e7e3d31b" path="/var/lib/kubelet/pods/63363952-85bf-44ba-a26b-0c19e7e3d31b/volumes"
Jul 13 18:55:13.849245 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:55:13.849210    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2bscc_2f0655bd-4a09-4314-8c1b-e30191eaec0e/ovn-acl-logging/0.log"
Jul 13 18:55:13.856511 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:55:13.856485    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2bscc_2f0655bd-4a09-4314-8c1b-e30191eaec0e/ovn-acl-logging/0.log"
Jul 13 18:55:13.857497 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:55:13.857476    2565 kubelet.go:1648] "Image garbage collection succeeded"
Jul 13 18:56:37.249005 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:56:37.248912    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-operator-77f648d694-6rspr"]
Jul 13 18:56:37.249584 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:56:37.249397    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="63363952-85bf-44ba-a26b-0c19e7e3d31b" containerName="console"
Jul 13 18:56:37.249584 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:56:37.249414    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="63363952-85bf-44ba-a26b-0c19e7e3d31b" containerName="console"
Jul 13 18:56:37.249584 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:56:37.249524    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="63363952-85bf-44ba-a26b-0c19e7e3d31b" containerName="console"
Jul 13 18:56:37.252423 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:56:37.252401    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-77f648d694-6rspr"
Jul 13 18:56:37.257953 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:56:37.257931    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-p6zqz\""
Jul 13 18:56:37.258228 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:56:37.258209    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Jul 13 18:56:37.258228 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:56:37.258224    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-certs\""
Jul 13 18:56:37.258415 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:56:37.258249    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Jul 13 18:56:37.258415 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:56:37.258289    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Jul 13 18:56:37.272007 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:56:37.271979    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Jul 13 18:56:37.273655 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:56:37.273632    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-77f648d694-6rspr"]
Jul 13 18:56:37.400521 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:56:37.400483    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/ff93ba62-c866-40bf-afd6-28e7fe5e3608-certificates\") pod \"keda-operator-77f648d694-6rspr\" (UID: \"ff93ba62-c866-40bf-afd6-28e7fe5e3608\") " pod="openshift-keda/keda-operator-77f648d694-6rspr"
Jul 13 18:56:37.400733 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:56:37.400551    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/ff93ba62-c866-40bf-afd6-28e7fe5e3608-cabundle0\") pod \"keda-operator-77f648d694-6rspr\" (UID: \"ff93ba62-c866-40bf-afd6-28e7fe5e3608\") " pod="openshift-keda/keda-operator-77f648d694-6rspr"
Jul 13 18:56:37.400733 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:56:37.400610    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lfd5\" (UniqueName: \"kubernetes.io/projected/ff93ba62-c866-40bf-afd6-28e7fe5e3608-kube-api-access-8lfd5\") pod \"keda-operator-77f648d694-6rspr\" (UID: \"ff93ba62-c866-40bf-afd6-28e7fe5e3608\") " pod="openshift-keda/keda-operator-77f648d694-6rspr"
Jul 13 18:56:37.502076 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:56:37.501972    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/ff93ba62-c866-40bf-afd6-28e7fe5e3608-certificates\") pod \"keda-operator-77f648d694-6rspr\" (UID: \"ff93ba62-c866-40bf-afd6-28e7fe5e3608\") " pod="openshift-keda/keda-operator-77f648d694-6rspr"
Jul 13 18:56:37.502076 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:56:37.502037    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/ff93ba62-c866-40bf-afd6-28e7fe5e3608-cabundle0\") pod \"keda-operator-77f648d694-6rspr\" (UID: \"ff93ba62-c866-40bf-afd6-28e7fe5e3608\") " pod="openshift-keda/keda-operator-77f648d694-6rspr"
Jul 13 18:56:37.502327 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:56:37.502086    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lfd5\" (UniqueName: \"kubernetes.io/projected/ff93ba62-c866-40bf-afd6-28e7fe5e3608-kube-api-access-8lfd5\") pod \"keda-operator-77f648d694-6rspr\" (UID: \"ff93ba62-c866-40bf-afd6-28e7fe5e3608\") " pod="openshift-keda/keda-operator-77f648d694-6rspr"
Jul 13 18:56:37.502327 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:56:37.502143    2565 projected.go:266] Couldn't get secret openshift-keda/keda-operator-certs: secret "keda-operator-certs" not found
Jul 13 18:56:37.502327 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:56:37.502170    2565 secret.go:281] references non-existent secret key: ca.crt
Jul 13 18:56:37.502327 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:56:37.502180    2565 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
Jul 13 18:56:37.502327 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:56:37.502195    2565 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-77f648d694-6rspr: [secret "keda-operator-certs" not found, references non-existent secret key: ca.crt]
Jul 13 18:56:37.502327 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:56:37.502273    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ff93ba62-c866-40bf-afd6-28e7fe5e3608-certificates podName:ff93ba62-c866-40bf-afd6-28e7fe5e3608 nodeName:}" failed. No retries permitted until 2026-07-13 18:56:38.002252349 +0000 UTC m=+384.527385742 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/ff93ba62-c866-40bf-afd6-28e7fe5e3608-certificates") pod "keda-operator-77f648d694-6rspr" (UID: "ff93ba62-c866-40bf-afd6-28e7fe5e3608") : [secret "keda-operator-certs" not found, references non-existent secret key: ca.crt]
Jul 13 18:56:37.502789 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:56:37.502769    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/ff93ba62-c866-40bf-afd6-28e7fe5e3608-cabundle0\") pod \"keda-operator-77f648d694-6rspr\" (UID: \"ff93ba62-c866-40bf-afd6-28e7fe5e3608\") " pod="openshift-keda/keda-operator-77f648d694-6rspr"
Jul 13 18:56:37.517231 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:56:37.517196    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lfd5\" (UniqueName: \"kubernetes.io/projected/ff93ba62-c866-40bf-afd6-28e7fe5e3608-kube-api-access-8lfd5\") pod \"keda-operator-77f648d694-6rspr\" (UID: \"ff93ba62-c866-40bf-afd6-28e7fe5e3608\") " pod="openshift-keda/keda-operator-77f648d694-6rspr"
Jul 13 18:56:38.005976 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:56:38.005935    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/ff93ba62-c866-40bf-afd6-28e7fe5e3608-certificates\") pod \"keda-operator-77f648d694-6rspr\" (UID: \"ff93ba62-c866-40bf-afd6-28e7fe5e3608\") " pod="openshift-keda/keda-operator-77f648d694-6rspr"
Jul 13 18:56:38.008945 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:56:38.008917    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/ff93ba62-c866-40bf-afd6-28e7fe5e3608-certificates\") pod \"keda-operator-77f648d694-6rspr\" (UID: \"ff93ba62-c866-40bf-afd6-28e7fe5e3608\") " pod="openshift-keda/keda-operator-77f648d694-6rspr"
Jul 13 18:56:38.162570 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:56:38.162519    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-77f648d694-6rspr"
Jul 13 18:56:38.331353 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:56:38.331321    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-77f648d694-6rspr"]
Jul 13 18:56:38.335763 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:56:38.335738    2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 13 18:56:38.806289 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:56:38.806254    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-77f648d694-6rspr" event={"ID":"ff93ba62-c866-40bf-afd6-28e7fe5e3608","Type":"ContainerStarted","Data":"30dbb5f573e89b49c3e923362b9b2f7b98757fcd497862fb81e8f46cca8b17ae"}
Jul 13 18:56:43.835079 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:56:43.835039    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-77f648d694-6rspr" event={"ID":"ff93ba62-c866-40bf-afd6-28e7fe5e3608","Type":"ContainerStarted","Data":"28efa6d47ca78862af8854d2cea29e186980d7bb3fea8faa6dd61e06ede2fdf6"}
Jul 13 18:56:43.835465 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:56:43.835174    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-operator-77f648d694-6rspr"
Jul 13 18:56:43.854745 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:56:43.854676    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-operator-77f648d694-6rspr" podStartSLOduration=1.540516741 podStartE2EDuration="6.854656736s" totalImagesPullingTime="5.314139995s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 18:56:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:56:43.853675814 +0000 UTC m=+390.378809252" watchObservedRunningTime="2026-07-13 18:56:43.854656736 +0000 UTC m=+390.379790152"
Jul 13 18:57:04.842606 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:57:04.842575    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-operator-77f648d694-6rspr"
Jul 13 18:57:44.781151 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:57:44.781063    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-6b69b6d547-f58tf"]
Jul 13 18:57:44.783327 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:57:44.783311    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-6b69b6d547-f58tf"
Jul 13 18:57:44.785764 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:57:44.785741    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-webhook-server-cert\""
Jul 13 18:57:44.786879 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:57:44.786849    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 13 18:57:44.786879 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:57:44.786872    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 13 18:57:44.787046 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:57:44.786929    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-controller-manager-dockercfg-4cvbg\""
Jul 13 18:57:44.795676 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:57:44.795652    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-6b69b6d547-f58tf"]
Jul 13 18:57:44.822746 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:57:44.822690    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-64568bcd49-5tglb"]
Jul 13 18:57:44.825653 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:57:44.825632    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-5tglb"
Jul 13 18:57:44.828002 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:57:44.827985    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-gwcfs\""
Jul 13 18:57:44.828112 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:57:44.827983    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Jul 13 18:57:44.836897 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:57:44.836864    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-5tglb"]
Jul 13 18:57:44.895034 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:57:44.894983    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hlcz9\" (UniqueName: \"kubernetes.io/projected/b1721a10-9182-4f1e-9146-f546ca94fed6-kube-api-access-hlcz9\") pod \"kserve-controller-manager-6b69b6d547-f58tf\" (UID: \"b1721a10-9182-4f1e-9146-f546ca94fed6\") " pod="kserve/kserve-controller-manager-6b69b6d547-f58tf"
Jul 13 18:57:44.895034 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:57:44.895038    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b1721a10-9182-4f1e-9146-f546ca94fed6-cert\") pod \"kserve-controller-manager-6b69b6d547-f58tf\" (UID: \"b1721a10-9182-4f1e-9146-f546ca94fed6\") " pod="kserve/kserve-controller-manager-6b69b6d547-f58tf"
Jul 13 18:57:44.995875 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:57:44.995833    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zbwf\" (UniqueName: \"kubernetes.io/projected/e2a6c28c-da28-464e-8636-4f5b1cafcefc-kube-api-access-7zbwf\") pod \"seaweedfs-64568bcd49-5tglb\" (UID: \"e2a6c28c-da28-464e-8636-4f5b1cafcefc\") " pod="kserve/seaweedfs-64568bcd49-5tglb"
Jul 13 18:57:44.996051 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:57:44.995927    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/e2a6c28c-da28-464e-8636-4f5b1cafcefc-data\") pod \"seaweedfs-64568bcd49-5tglb\" (UID: \"e2a6c28c-da28-464e-8636-4f5b1cafcefc\") " pod="kserve/seaweedfs-64568bcd49-5tglb"
Jul 13 18:57:44.996051 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:57:44.996033    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlcz9\" (UniqueName: \"kubernetes.io/projected/b1721a10-9182-4f1e-9146-f546ca94fed6-kube-api-access-hlcz9\") pod \"kserve-controller-manager-6b69b6d547-f58tf\" (UID: \"b1721a10-9182-4f1e-9146-f546ca94fed6\") " pod="kserve/kserve-controller-manager-6b69b6d547-f58tf"
Jul 13 18:57:44.996143 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:57:44.996064    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b1721a10-9182-4f1e-9146-f546ca94fed6-cert\") pod \"kserve-controller-manager-6b69b6d547-f58tf\" (UID: \"b1721a10-9182-4f1e-9146-f546ca94fed6\") " pod="kserve/kserve-controller-manager-6b69b6d547-f58tf"
Jul 13 18:57:44.996222 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:57:44.996208    2565 secret.go:189] Couldn't get secret kserve/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found
Jul 13 18:57:44.996280 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:57:44.996270    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1721a10-9182-4f1e-9146-f546ca94fed6-cert podName:b1721a10-9182-4f1e-9146-f546ca94fed6 nodeName:}" failed. No retries permitted until 2026-07-13 18:57:45.496247613 +0000 UTC m=+452.021381011 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/b1721a10-9182-4f1e-9146-f546ca94fed6-cert") pod "kserve-controller-manager-6b69b6d547-f58tf" (UID: "b1721a10-9182-4f1e-9146-f546ca94fed6") : secret "kserve-webhook-server-cert" not found
Jul 13 18:57:45.009306 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:57:45.009270    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hlcz9\" (UniqueName: \"kubernetes.io/projected/b1721a10-9182-4f1e-9146-f546ca94fed6-kube-api-access-hlcz9\") pod \"kserve-controller-manager-6b69b6d547-f58tf\" (UID: \"b1721a10-9182-4f1e-9146-f546ca94fed6\") " pod="kserve/kserve-controller-manager-6b69b6d547-f58tf"
Jul 13 18:57:45.096694 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:57:45.096576    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zbwf\" (UniqueName: \"kubernetes.io/projected/e2a6c28c-da28-464e-8636-4f5b1cafcefc-kube-api-access-7zbwf\") pod \"seaweedfs-64568bcd49-5tglb\" (UID: \"e2a6c28c-da28-464e-8636-4f5b1cafcefc\") " pod="kserve/seaweedfs-64568bcd49-5tglb"
Jul 13 18:57:45.096694 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:57:45.096683    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/e2a6c28c-da28-464e-8636-4f5b1cafcefc-data\") pod \"seaweedfs-64568bcd49-5tglb\" (UID: \"e2a6c28c-da28-464e-8636-4f5b1cafcefc\") " pod="kserve/seaweedfs-64568bcd49-5tglb"
Jul 13 18:57:45.097082 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:57:45.097060    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/e2a6c28c-da28-464e-8636-4f5b1cafcefc-data\") pod \"seaweedfs-64568bcd49-5tglb\" (UID: \"e2a6c28c-da28-464e-8636-4f5b1cafcefc\") " pod="kserve/seaweedfs-64568bcd49-5tglb"
Jul 13 18:57:45.105309 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:57:45.105279    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zbwf\" (UniqueName: \"kubernetes.io/projected/e2a6c28c-da28-464e-8636-4f5b1cafcefc-kube-api-access-7zbwf\") pod \"seaweedfs-64568bcd49-5tglb\" (UID: \"e2a6c28c-da28-464e-8636-4f5b1cafcefc\") " pod="kserve/seaweedfs-64568bcd49-5tglb"
Jul 13 18:57:45.136130 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:57:45.136087    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-5tglb"
Jul 13 18:57:45.282395 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:57:45.282345    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-5tglb"]
Jul 13 18:57:45.285044 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:57:45.285006    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2a6c28c_da28_464e_8636_4f5b1cafcefc.slice/crio-4870d52218067a450c3265de0fb47d06166921fd52f3515f1cba8ee6a2826c7d WatchSource:0}: Error finding container 4870d52218067a450c3265de0fb47d06166921fd52f3515f1cba8ee6a2826c7d: Status 404 returned error can't find the container with id 4870d52218067a450c3265de0fb47d06166921fd52f3515f1cba8ee6a2826c7d
Jul 13 18:57:45.500778 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:57:45.500662    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b1721a10-9182-4f1e-9146-f546ca94fed6-cert\") pod \"kserve-controller-manager-6b69b6d547-f58tf\" (UID: \"b1721a10-9182-4f1e-9146-f546ca94fed6\") " pod="kserve/kserve-controller-manager-6b69b6d547-f58tf"
Jul 13 18:57:45.503282 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:57:45.503248    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b1721a10-9182-4f1e-9146-f546ca94fed6-cert\") pod \"kserve-controller-manager-6b69b6d547-f58tf\" (UID: \"b1721a10-9182-4f1e-9146-f546ca94fed6\") " pod="kserve/kserve-controller-manager-6b69b6d547-f58tf"
Jul 13 18:57:45.695467 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:57:45.695429    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-6b69b6d547-f58tf"
Jul 13 18:57:45.844305 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:57:45.844274    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-6b69b6d547-f58tf"]
Jul 13 18:57:45.846264 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:57:45.846233    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1721a10_9182_4f1e_9146_f546ca94fed6.slice/crio-14e9eadcccb0ac65a952a071cdc5985f79d591a9422f0b70d85d5fbc378e0344 WatchSource:0}: Error finding container 14e9eadcccb0ac65a952a071cdc5985f79d591a9422f0b70d85d5fbc378e0344: Status 404 returned error can't find the container with id 14e9eadcccb0ac65a952a071cdc5985f79d591a9422f0b70d85d5fbc378e0344
Jul 13 18:57:46.104735 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:57:46.104617    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-6b69b6d547-f58tf" event={"ID":"b1721a10-9182-4f1e-9146-f546ca94fed6","Type":"ContainerStarted","Data":"14e9eadcccb0ac65a952a071cdc5985f79d591a9422f0b70d85d5fbc378e0344"}
Jul 13 18:57:46.105770 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:57:46.105746    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-5tglb" event={"ID":"e2a6c28c-da28-464e-8636-4f5b1cafcefc","Type":"ContainerStarted","Data":"4870d52218067a450c3265de0fb47d06166921fd52f3515f1cba8ee6a2826c7d"}
Jul 13 18:57:51.134365 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:57:51.134231    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-5tglb" event={"ID":"e2a6c28c-da28-464e-8636-4f5b1cafcefc","Type":"ContainerStarted","Data":"6ba7703e65d17a71ef8a231849b8b187d9200d7efc6615f0c80824f05ce22a2f"}
Jul 13 18:57:51.134365 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:57:51.134319    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-64568bcd49-5tglb"
Jul 13 18:57:51.135787 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:57:51.135764    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-6b69b6d547-f58tf" event={"ID":"b1721a10-9182-4f1e-9146-f546ca94fed6","Type":"ContainerStarted","Data":"8432ca785e1647d1c706aa41386cd59354b8fada83f39bc4af588198ded502c4"}
Jul 13 18:57:51.135878 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:57:51.135813    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-6b69b6d547-f58tf"
Jul 13 18:57:51.151054 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:57:51.151004    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-64568bcd49-5tglb" podStartSLOduration=1.50120287 podStartE2EDuration="7.150988745s" totalImagesPullingTime="5.649785875s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 18:57:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:57:51.150117762 +0000 UTC m=+457.675251178" watchObservedRunningTime="2026-07-13 18:57:51.150988745 +0000 UTC m=+457.676122161"
Jul 13 18:57:51.167066 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:57:51.167011    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-6b69b6d547-f58tf" podStartSLOduration=2.130986794 podStartE2EDuration="7.16699631s" totalImagesPullingTime="5.036009516s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 18:57:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:57:51.165323615 +0000 UTC m=+457.690457031" watchObservedRunningTime="2026-07-13 18:57:51.16699631 +0000 UTC m=+457.692129725"
Jul 13 18:57:57.143115 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:57:57.143083    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-64568bcd49-5tglb"
Jul 13 18:58:15.318235 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:58:15.318198    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-8bbbbb749-r68gb"]
Jul 13 18:58:15.323194 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:58:15.323171    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8bbbbb749-r68gb"
Jul 13 18:58:15.336147 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:58:15.336122    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-8bbbbb749-r68gb"]
Jul 13 18:58:15.463611 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:58:15.463564    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4f5c11d7-4b13-401a-bbec-9b9516fd5cda-service-ca\") pod \"console-8bbbbb749-r68gb\" (UID: \"4f5c11d7-4b13-401a-bbec-9b9516fd5cda\") " pod="openshift-console/console-8bbbbb749-r68gb"
Jul 13 18:58:15.463814 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:58:15.463628    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tww8t\" (UniqueName: \"kubernetes.io/projected/4f5c11d7-4b13-401a-bbec-9b9516fd5cda-kube-api-access-tww8t\") pod \"console-8bbbbb749-r68gb\" (UID: \"4f5c11d7-4b13-401a-bbec-9b9516fd5cda\") " pod="openshift-console/console-8bbbbb749-r68gb"
Jul 13 18:58:15.463814 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:58:15.463661    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4f5c11d7-4b13-401a-bbec-9b9516fd5cda-console-config\") pod \"console-8bbbbb749-r68gb\" (UID: \"4f5c11d7-4b13-401a-bbec-9b9516fd5cda\") " pod="openshift-console/console-8bbbbb749-r68gb"
Jul 13 18:58:15.463814 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:58:15.463682    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4f5c11d7-4b13-401a-bbec-9b9516fd5cda-oauth-serving-cert\") pod \"console-8bbbbb749-r68gb\" (UID: \"4f5c11d7-4b13-401a-bbec-9b9516fd5cda\") " pod="openshift-console/console-8bbbbb749-r68gb"
Jul 13 18:58:15.463814 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:58:15.463730    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4f5c11d7-4b13-401a-bbec-9b9516fd5cda-console-oauth-config\") pod \"console-8bbbbb749-r68gb\" (UID: \"4f5c11d7-4b13-401a-bbec-9b9516fd5cda\") " pod="openshift-console/console-8bbbbb749-r68gb"
Jul 13 18:58:15.463814 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:58:15.463757    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4f5c11d7-4b13-401a-bbec-9b9516fd5cda-console-serving-cert\") pod \"console-8bbbbb749-r68gb\" (UID: \"4f5c11d7-4b13-401a-bbec-9b9516fd5cda\") " pod="openshift-console/console-8bbbbb749-r68gb"
Jul 13 18:58:15.463814 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:58:15.463775    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4f5c11d7-4b13-401a-bbec-9b9516fd5cda-trusted-ca-bundle\") pod \"console-8bbbbb749-r68gb\" (UID: \"4f5c11d7-4b13-401a-bbec-9b9516fd5cda\") " pod="openshift-console/console-8bbbbb749-r68gb"
Jul 13 18:58:15.564981 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:58:15.564944    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4f5c11d7-4b13-401a-bbec-9b9516fd5cda-service-ca\") pod \"console-8bbbbb749-r68gb\" (UID: \"4f5c11d7-4b13-401a-bbec-9b9516fd5cda\") " pod="openshift-console/console-8bbbbb749-r68gb"
Jul 13 18:58:15.565171 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:58:15.564996    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tww8t\" (UniqueName: \"kubernetes.io/projected/4f5c11d7-4b13-401a-bbec-9b9516fd5cda-kube-api-access-tww8t\") pod \"console-8bbbbb749-r68gb\" (UID: \"4f5c11d7-4b13-401a-bbec-9b9516fd5cda\") " pod="openshift-console/console-8bbbbb749-r68gb"
Jul 13 18:58:15.565171 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:58:15.565020    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4f5c11d7-4b13-401a-bbec-9b9516fd5cda-console-config\") pod \"console-8bbbbb749-r68gb\" (UID: \"4f5c11d7-4b13-401a-bbec-9b9516fd5cda\") " pod="openshift-console/console-8bbbbb749-r68gb"
Jul 13 18:58:15.565171 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:58:15.565046    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4f5c11d7-4b13-401a-bbec-9b9516fd5cda-oauth-serving-cert\") pod \"console-8bbbbb749-r68gb\" (UID: \"4f5c11d7-4b13-401a-bbec-9b9516fd5cda\") " pod="openshift-console/console-8bbbbb749-r68gb"
Jul 13 18:58:15.565171 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:58:15.565067    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4f5c11d7-4b13-401a-bbec-9b9516fd5cda-console-oauth-config\") pod \"console-8bbbbb749-r68gb\" (UID: \"4f5c11d7-4b13-401a-bbec-9b9516fd5cda\") " pod="openshift-console/console-8bbbbb749-r68gb"
Jul 13 18:58:15.565171 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:58:15.565088    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4f5c11d7-4b13-401a-bbec-9b9516fd5cda-console-serving-cert\") pod \"console-8bbbbb749-r68gb\" (UID: \"4f5c11d7-4b13-401a-bbec-9b9516fd5cda\") " pod="openshift-console/console-8bbbbb749-r68gb"
Jul 13 18:58:15.565171 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:58:15.565112    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4f5c11d7-4b13-401a-bbec-9b9516fd5cda-trusted-ca-bundle\") pod \"console-8bbbbb749-r68gb\" (UID: \"4f5c11d7-4b13-401a-bbec-9b9516fd5cda\") " pod="openshift-console/console-8bbbbb749-r68gb"
Jul 13 18:58:15.565943 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:58:15.565872    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4f5c11d7-4b13-401a-bbec-9b9516fd5cda-console-config\") pod \"console-8bbbbb749-r68gb\" (UID: \"4f5c11d7-4b13-401a-bbec-9b9516fd5cda\") " pod="openshift-console/console-8bbbbb749-r68gb"
Jul 13 18:58:15.565943 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:58:15.565881    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4f5c11d7-4b13-401a-bbec-9b9516fd5cda-oauth-serving-cert\") pod \"console-8bbbbb749-r68gb\" (UID: \"4f5c11d7-4b13-401a-bbec-9b9516fd5cda\") " pod="openshift-console/console-8bbbbb749-r68gb"
Jul 13 18:58:15.566112 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:58:15.566005    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4f5c11d7-4b13-401a-bbec-9b9516fd5cda-trusted-ca-bundle\") pod \"console-8bbbbb749-r68gb\" (UID: \"4f5c11d7-4b13-401a-bbec-9b9516fd5cda\") " pod="openshift-console/console-8bbbbb749-r68gb"
Jul 13 18:58:15.566112 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:58:15.566077    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4f5c11d7-4b13-401a-bbec-9b9516fd5cda-service-ca\") pod \"console-8bbbbb749-r68gb\" (UID: \"4f5c11d7-4b13-401a-bbec-9b9516fd5cda\") " pod="openshift-console/console-8bbbbb749-r68gb"
Jul 13 18:58:15.567663 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:58:15.567641    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4f5c11d7-4b13-401a-bbec-9b9516fd5cda-console-oauth-config\") pod \"console-8bbbbb749-r68gb\" (UID: \"4f5c11d7-4b13-401a-bbec-9b9516fd5cda\") " pod="openshift-console/console-8bbbbb749-r68gb"
Jul 13 18:58:15.567864 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:58:15.567846    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4f5c11d7-4b13-401a-bbec-9b9516fd5cda-console-serving-cert\") pod \"console-8bbbbb749-r68gb\" (UID: \"4f5c11d7-4b13-401a-bbec-9b9516fd5cda\") " pod="openshift-console/console-8bbbbb749-r68gb"
Jul 13 18:58:15.573602 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:58:15.573546    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tww8t\" (UniqueName: \"kubernetes.io/projected/4f5c11d7-4b13-401a-bbec-9b9516fd5cda-kube-api-access-tww8t\") pod \"console-8bbbbb749-r68gb\" (UID: \"4f5c11d7-4b13-401a-bbec-9b9516fd5cda\") " pod="openshift-console/console-8bbbbb749-r68gb"
Jul 13 18:58:15.632243 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:58:15.632209    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8bbbbb749-r68gb"
Jul 13 18:58:15.776395 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:58:15.776371    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-8bbbbb749-r68gb"]
Jul 13 18:58:15.778895 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:58:15.778862    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f5c11d7_4b13_401a_bbec_9b9516fd5cda.slice/crio-4a3df8d0a0b60f777231ed0c7febf59bb41419d79481adf878ddc26448b5e73d WatchSource:0}: Error finding container 4a3df8d0a0b60f777231ed0c7febf59bb41419d79481adf878ddc26448b5e73d: Status 404 returned error can't find the container with id 4a3df8d0a0b60f777231ed0c7febf59bb41419d79481adf878ddc26448b5e73d
Jul 13 18:58:16.251666 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:58:16.251583    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8bbbbb749-r68gb" event={"ID":"4f5c11d7-4b13-401a-bbec-9b9516fd5cda","Type":"ContainerStarted","Data":"2d348de00b7fbd799a5aa107eff8fe3653d328b55ebbb6c94bc596e68cd2d59c"}
Jul 13 18:58:16.251666 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:58:16.251628    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8bbbbb749-r68gb" event={"ID":"4f5c11d7-4b13-401a-bbec-9b9516fd5cda","Type":"ContainerStarted","Data":"4a3df8d0a0b60f777231ed0c7febf59bb41419d79481adf878ddc26448b5e73d"}
Jul 13 18:58:16.271430 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:58:16.271379    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-8bbbbb749-r68gb" podStartSLOduration=1.2713642969999999 podStartE2EDuration="1.271364297s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 18:58:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:58:16.26952801 +0000 UTC m=+482.794661449" watchObservedRunningTime="2026-07-13 18:58:16.271364297 +0000 UTC m=+482.796497713"
Jul 13 18:58:22.145665 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:58:22.145625    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-6b69b6d547-f58tf"
Jul 13 18:58:25.633030 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:58:25.632990    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-8bbbbb749-r68gb"
Jul 13 18:58:25.633445 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:58:25.633095    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-8bbbbb749-r68gb"
Jul 13 18:58:25.637995 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:58:25.637970    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-8bbbbb749-r68gb"
Jul 13 18:58:26.304411 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:58:26.304382    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-8bbbbb749-r68gb"
Jul 13 18:58:26.349083 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:58:26.349049    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-95b84c78f-ggrgb"]
Jul 13 18:58:35.496480 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:58:35.496443    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-cg69l"]
Jul 13 18:58:35.499834 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:58:35.499816    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-cg69l"
Jul 13 18:58:35.502580 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:58:35.502558    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-smthj\""
Jul 13 18:58:35.502761 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:58:35.502559    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Jul 13 18:58:35.502761 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:58:35.502665    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Jul 13 18:58:35.516518 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:58:35.516362    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-cg69l"]
Jul 13 18:58:35.547896 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:58:35.547857    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48gpp\" (UniqueName: \"kubernetes.io/projected/920c6821-87f1-4599-b0e6-0b897865abec-kube-api-access-48gpp\") pod \"cert-manager-operator-controller-manager-684f79c9b9-cg69l\" (UID: \"920c6821-87f1-4599-b0e6-0b897865abec\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-cg69l"
Jul 13 18:58:35.548099 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:58:35.547975    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/920c6821-87f1-4599-b0e6-0b897865abec-tmp\") pod \"cert-manager-operator-controller-manager-684f79c9b9-cg69l\" (UID: \"920c6821-87f1-4599-b0e6-0b897865abec\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-cg69l"
Jul 13 18:58:35.649390 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:58:35.649349    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/920c6821-87f1-4599-b0e6-0b897865abec-tmp\") pod \"cert-manager-operator-controller-manager-684f79c9b9-cg69l\" (UID: \"920c6821-87f1-4599-b0e6-0b897865abec\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-cg69l"
Jul 13 18:58:35.649568 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:58:35.649406    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48gpp\" (UniqueName: \"kubernetes.io/projected/920c6821-87f1-4599-b0e6-0b897865abec-kube-api-access-48gpp\") pod \"cert-manager-operator-controller-manager-684f79c9b9-cg69l\" (UID: \"920c6821-87f1-4599-b0e6-0b897865abec\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-cg69l"
Jul 13 18:58:35.649792 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:58:35.649771    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/920c6821-87f1-4599-b0e6-0b897865abec-tmp\") pod \"cert-manager-operator-controller-manager-684f79c9b9-cg69l\" (UID: \"920c6821-87f1-4599-b0e6-0b897865abec\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-cg69l"
Jul 13 18:58:35.659162 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:58:35.659128    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-48gpp\" (UniqueName: \"kubernetes.io/projected/920c6821-87f1-4599-b0e6-0b897865abec-kube-api-access-48gpp\") pod \"cert-manager-operator-controller-manager-684f79c9b9-cg69l\" (UID: \"920c6821-87f1-4599-b0e6-0b897865abec\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-cg69l"
Jul 13 18:58:35.810469 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:58:35.810430    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-cg69l"
Jul 13 18:58:35.983996 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:58:35.983962    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-cg69l"]
Jul 13 18:58:35.988459 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:58:35.988430    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod920c6821_87f1_4599_b0e6_0b897865abec.slice/crio-73d2db0aba874c76314f405faa1f8329c6ada067fd0ffaec5dc14276c2b9c4dc WatchSource:0}: Error finding container 73d2db0aba874c76314f405faa1f8329c6ada067fd0ffaec5dc14276c2b9c4dc: Status 404 returned error can't find the container with id 73d2db0aba874c76314f405faa1f8329c6ada067fd0ffaec5dc14276c2b9c4dc
Jul 13 18:58:36.346613 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:58:36.346577    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-cg69l" event={"ID":"920c6821-87f1-4599-b0e6-0b897865abec","Type":"ContainerStarted","Data":"73d2db0aba874c76314f405faa1f8329c6ada067fd0ffaec5dc14276c2b9c4dc"}
Jul 13 18:58:38.358048 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:58:38.358014    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-cg69l" event={"ID":"920c6821-87f1-4599-b0e6-0b897865abec","Type":"ContainerStarted","Data":"6ab5c135750c935efca1edb88b614cb0873de250f0be9593a8a9920ee8ccf276"}
Jul 13 18:58:38.381340 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:58:38.381280    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-cg69l" podStartSLOduration=1.5375717500000001 podStartE2EDuration="3.38126283s" totalImagesPullingTime="1.84369108s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 18:58:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:58:38.378669565 +0000 UTC m=+504.903802981" watchObservedRunningTime="2026-07-13 18:58:38.38126283 +0000 UTC m=+504.906396245"
Jul 13 18:58:50.263245 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:58:50.263202    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-mqcj2"]
Jul 13 18:58:50.266881 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:58:50.266861    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-mqcj2"
Jul 13 18:58:50.269994 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:58:50.269967    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Jul 13 18:58:50.269994 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:58:50.269987    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Jul 13 18:58:50.270915 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:58:50.270896    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-fh5l8\""
Jul 13 18:58:50.276985 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:58:50.276959    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-mqcj2"]
Jul 13 18:58:50.383909 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:58:50.383860    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/30cd87db-85cb-4782-b734-06e6cf22fbd7-bound-sa-token\") pod \"cert-manager-55b4d954d5-mqcj2\" (UID: \"30cd87db-85cb-4782-b734-06e6cf22fbd7\") " pod="cert-manager/cert-manager-55b4d954d5-mqcj2"
Jul 13 18:58:50.384101 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:58:50.383938    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkk98\" (UniqueName: \"kubernetes.io/projected/30cd87db-85cb-4782-b734-06e6cf22fbd7-kube-api-access-zkk98\") pod \"cert-manager-55b4d954d5-mqcj2\" (UID: \"30cd87db-85cb-4782-b734-06e6cf22fbd7\") " pod="cert-manager/cert-manager-55b4d954d5-mqcj2"
Jul 13 18:58:50.485200 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:58:50.485161    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/30cd87db-85cb-4782-b734-06e6cf22fbd7-bound-sa-token\") pod \"cert-manager-55b4d954d5-mqcj2\" (UID: \"30cd87db-85cb-4782-b734-06e6cf22fbd7\") " pod="cert-manager/cert-manager-55b4d954d5-mqcj2"
Jul 13 18:58:50.485412 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:58:50.485239    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkk98\" (UniqueName: \"kubernetes.io/projected/30cd87db-85cb-4782-b734-06e6cf22fbd7-kube-api-access-zkk98\") pod \"cert-manager-55b4d954d5-mqcj2\" (UID: \"30cd87db-85cb-4782-b734-06e6cf22fbd7\") " pod="cert-manager/cert-manager-55b4d954d5-mqcj2"
Jul 13 18:58:50.494275 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:58:50.494247    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkk98\" (UniqueName: \"kubernetes.io/projected/30cd87db-85cb-4782-b734-06e6cf22fbd7-kube-api-access-zkk98\") pod \"cert-manager-55b4d954d5-mqcj2\" (UID: \"30cd87db-85cb-4782-b734-06e6cf22fbd7\") " pod="cert-manager/cert-manager-55b4d954d5-mqcj2"
Jul 13 18:58:50.494376 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:58:50.494247    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/30cd87db-85cb-4782-b734-06e6cf22fbd7-bound-sa-token\") pod \"cert-manager-55b4d954d5-mqcj2\" (UID: \"30cd87db-85cb-4782-b734-06e6cf22fbd7\") " pod="cert-manager/cert-manager-55b4d954d5-mqcj2"
Jul 13 18:58:50.586127 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:58:50.586041    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-mqcj2"
Jul 13 18:58:50.736066 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:58:50.736030    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-mqcj2"]
Jul 13 18:58:50.739041 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:58:50.739011    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod30cd87db_85cb_4782_b734_06e6cf22fbd7.slice/crio-8561a8913d451cb8d84db920bf26bd96f12eb58bb91d6d9edbd716ee61e3753e WatchSource:0}: Error finding container 8561a8913d451cb8d84db920bf26bd96f12eb58bb91d6d9edbd716ee61e3753e: Status 404 returned error can't find the container with id 8561a8913d451cb8d84db920bf26bd96f12eb58bb91d6d9edbd716ee61e3753e
Jul 13 18:58:51.372263 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:58:51.372226    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-95b84c78f-ggrgb" podUID="030384be-1be3-4bd0-8ba2-7cf6df3ba223" containerName="console" containerID="cri-o://091f84aa260dcbc8789700c057e13e8ed14b00e1e8f3074f2329ba2407265a9f" gracePeriod=15
Jul 13 18:58:51.420859 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:58:51.420824    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-mqcj2" event={"ID":"30cd87db-85cb-4782-b734-06e6cf22fbd7","Type":"ContainerStarted","Data":"8561a8913d451cb8d84db920bf26bd96f12eb58bb91d6d9edbd716ee61e3753e"}
Jul 13 18:58:51.649265 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:58:51.649234    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-95b84c78f-ggrgb"
Jul 13 18:58:51.696040 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:58:51.696001    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/030384be-1be3-4bd0-8ba2-7cf6df3ba223-console-oauth-config\") pod \"030384be-1be3-4bd0-8ba2-7cf6df3ba223\" (UID: \"030384be-1be3-4bd0-8ba2-7cf6df3ba223\") "
Jul 13 18:58:51.696235 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:58:51.696061    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/030384be-1be3-4bd0-8ba2-7cf6df3ba223-service-ca\") pod \"030384be-1be3-4bd0-8ba2-7cf6df3ba223\" (UID: \"030384be-1be3-4bd0-8ba2-7cf6df3ba223\") "
Jul 13 18:58:51.696235 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:58:51.696090    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/030384be-1be3-4bd0-8ba2-7cf6df3ba223-console-serving-cert\") pod \"030384be-1be3-4bd0-8ba2-7cf6df3ba223\" (UID: \"030384be-1be3-4bd0-8ba2-7cf6df3ba223\") "
Jul 13 18:58:51.696235 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:58:51.696115    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/030384be-1be3-4bd0-8ba2-7cf6df3ba223-trusted-ca-bundle\") pod \"030384be-1be3-4bd0-8ba2-7cf6df3ba223\" (UID: \"030384be-1be3-4bd0-8ba2-7cf6df3ba223\") "
Jul 13 18:58:51.696235 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:58:51.696181    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jkxfx\" (UniqueName: \"kubernetes.io/projected/030384be-1be3-4bd0-8ba2-7cf6df3ba223-kube-api-access-jkxfx\") pod \"030384be-1be3-4bd0-8ba2-7cf6df3ba223\" (UID: \"030384be-1be3-4bd0-8ba2-7cf6df3ba223\") "
Jul 13 18:58:51.696235 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:58:51.696226    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/030384be-1be3-4bd0-8ba2-7cf6df3ba223-oauth-serving-cert\") pod \"030384be-1be3-4bd0-8ba2-7cf6df3ba223\" (UID: \"030384be-1be3-4bd0-8ba2-7cf6df3ba223\") "
Jul 13 18:58:51.696500 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:58:51.696260    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/030384be-1be3-4bd0-8ba2-7cf6df3ba223-console-config\") pod \"030384be-1be3-4bd0-8ba2-7cf6df3ba223\" (UID: \"030384be-1be3-4bd0-8ba2-7cf6df3ba223\") "
Jul 13 18:58:51.696622 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:58:51.696578    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/030384be-1be3-4bd0-8ba2-7cf6df3ba223-service-ca" (OuterVolumeSpecName: "service-ca") pod "030384be-1be3-4bd0-8ba2-7cf6df3ba223" (UID: "030384be-1be3-4bd0-8ba2-7cf6df3ba223"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 18:58:51.696883 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:58:51.696857    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/030384be-1be3-4bd0-8ba2-7cf6df3ba223-console-config" (OuterVolumeSpecName: "console-config") pod "030384be-1be3-4bd0-8ba2-7cf6df3ba223" (UID: "030384be-1be3-4bd0-8ba2-7cf6df3ba223"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 18:58:51.696972 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:58:51.696885    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/030384be-1be3-4bd0-8ba2-7cf6df3ba223-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "030384be-1be3-4bd0-8ba2-7cf6df3ba223" (UID: "030384be-1be3-4bd0-8ba2-7cf6df3ba223"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 18:58:51.697103 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:58:51.697078    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/030384be-1be3-4bd0-8ba2-7cf6df3ba223-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "030384be-1be3-4bd0-8ba2-7cf6df3ba223" (UID: "030384be-1be3-4bd0-8ba2-7cf6df3ba223"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 18:58:51.699041 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:58:51.699013    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/030384be-1be3-4bd0-8ba2-7cf6df3ba223-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "030384be-1be3-4bd0-8ba2-7cf6df3ba223" (UID: "030384be-1be3-4bd0-8ba2-7cf6df3ba223"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 18:58:51.699361 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:58:51.699323    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/030384be-1be3-4bd0-8ba2-7cf6df3ba223-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "030384be-1be3-4bd0-8ba2-7cf6df3ba223" (UID: "030384be-1be3-4bd0-8ba2-7cf6df3ba223"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 18:58:51.699616 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:58:51.699592    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/030384be-1be3-4bd0-8ba2-7cf6df3ba223-kube-api-access-jkxfx" (OuterVolumeSpecName: "kube-api-access-jkxfx") pod "030384be-1be3-4bd0-8ba2-7cf6df3ba223" (UID: "030384be-1be3-4bd0-8ba2-7cf6df3ba223"). InnerVolumeSpecName "kube-api-access-jkxfx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 18:58:51.797379 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:58:51.797336    2565 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/030384be-1be3-4bd0-8ba2-7cf6df3ba223-oauth-serving-cert\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 13 18:58:51.797379 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:58:51.797377    2565 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/030384be-1be3-4bd0-8ba2-7cf6df3ba223-console-config\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 13 18:58:51.797379 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:58:51.797389    2565 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/030384be-1be3-4bd0-8ba2-7cf6df3ba223-console-oauth-config\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 13 18:58:51.797625 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:58:51.797398    2565 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/030384be-1be3-4bd0-8ba2-7cf6df3ba223-service-ca\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 13 18:58:51.797625 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:58:51.797407    2565 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/030384be-1be3-4bd0-8ba2-7cf6df3ba223-console-serving-cert\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 13 18:58:51.797625 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:58:51.797414    2565 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/030384be-1be3-4bd0-8ba2-7cf6df3ba223-trusted-ca-bundle\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 13 18:58:51.797625 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:58:51.797423    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jkxfx\" (UniqueName: \"kubernetes.io/projected/030384be-1be3-4bd0-8ba2-7cf6df3ba223-kube-api-access-jkxfx\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 13 18:58:52.428646 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:58:52.428551    2565 generic.go:356] "Generic (PLEG): container finished" podID="030384be-1be3-4bd0-8ba2-7cf6df3ba223" containerID="091f84aa260dcbc8789700c057e13e8ed14b00e1e8f3074f2329ba2407265a9f" exitCode=0
Jul 13 18:58:52.428646 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:58:52.428598    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-95b84c78f-ggrgb" event={"ID":"030384be-1be3-4bd0-8ba2-7cf6df3ba223","Type":"ContainerDied","Data":"091f84aa260dcbc8789700c057e13e8ed14b00e1e8f3074f2329ba2407265a9f"}
Jul 13 18:58:52.428646 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:58:52.428636    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-95b84c78f-ggrgb"
Jul 13 18:58:52.429240 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:58:52.428658    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-95b84c78f-ggrgb" event={"ID":"030384be-1be3-4bd0-8ba2-7cf6df3ba223","Type":"ContainerDied","Data":"7b5f089b396a27edcadbc0a921db23be0ce01ffac174fadc593faa938e97e49b"}
Jul 13 18:58:52.429240 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:58:52.428682    2565 scope.go:117] "RemoveContainer" containerID="091f84aa260dcbc8789700c057e13e8ed14b00e1e8f3074f2329ba2407265a9f"
Jul 13 18:58:52.439285 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:58:52.439259    2565 scope.go:117] "RemoveContainer" containerID="091f84aa260dcbc8789700c057e13e8ed14b00e1e8f3074f2329ba2407265a9f"
Jul 13 18:58:52.439663 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:58:52.439633    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"091f84aa260dcbc8789700c057e13e8ed14b00e1e8f3074f2329ba2407265a9f\": container with ID starting with 091f84aa260dcbc8789700c057e13e8ed14b00e1e8f3074f2329ba2407265a9f not found: ID does not exist" containerID="091f84aa260dcbc8789700c057e13e8ed14b00e1e8f3074f2329ba2407265a9f"
Jul 13 18:58:52.439777 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:58:52.439670    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"091f84aa260dcbc8789700c057e13e8ed14b00e1e8f3074f2329ba2407265a9f"} err="failed to get container status \"091f84aa260dcbc8789700c057e13e8ed14b00e1e8f3074f2329ba2407265a9f\": rpc error: code = NotFound desc = could not find container \"091f84aa260dcbc8789700c057e13e8ed14b00e1e8f3074f2329ba2407265a9f\": container with ID starting with 091f84aa260dcbc8789700c057e13e8ed14b00e1e8f3074f2329ba2407265a9f not found: ID does not exist"
Jul 13 18:58:52.447091 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:58:52.447057    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-95b84c78f-ggrgb"]
Jul 13 18:58:52.454013 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:58:52.453982    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-95b84c78f-ggrgb"]
Jul 13 18:58:54.048036 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:58:54.047999    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="030384be-1be3-4bd0-8ba2-7cf6df3ba223" path="/var/lib/kubelet/pods/030384be-1be3-4bd0-8ba2-7cf6df3ba223/volumes"
Jul 13 18:58:54.440523 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:58:54.440424    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-mqcj2" event={"ID":"30cd87db-85cb-4782-b734-06e6cf22fbd7","Type":"ContainerStarted","Data":"660044369b31fa56423ccab323e0b82b3d7f1eb9e7fa7430fe118366bc85afb4"}
Jul 13 18:58:54.456523 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:58:54.456472    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-mqcj2" podStartSLOduration=1.747166692 podStartE2EDuration="4.456449421s" totalImagesPullingTime="2.709282729s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 18:58:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:58:54.456227659 +0000 UTC m=+520.981361076" watchObservedRunningTime="2026-07-13 18:58:54.456449421 +0000 UTC m=+520.981582838"
Jul 13 18:59:19.207638 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:59:19.207605    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-8g97n"]
Jul 13 18:59:19.208063 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:59:19.208016    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="030384be-1be3-4bd0-8ba2-7cf6df3ba223" containerName="console"
Jul 13 18:59:19.208063 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:59:19.208028    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="030384be-1be3-4bd0-8ba2-7cf6df3ba223" containerName="console"
Jul 13 18:59:19.208142 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:59:19.208110    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="030384be-1be3-4bd0-8ba2-7cf6df3ba223" containerName="console"
Jul 13 18:59:19.211292 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:59:19.211272    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-8g97n"
Jul 13 18:59:19.213952 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:59:19.213928    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 13 18:59:19.214081 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:59:19.214036    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-default-dockercfg-xtqm7\""
Jul 13 18:59:19.214436 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:59:19.214417    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 13 18:59:19.214590 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:59:19.214574    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 13 18:59:19.224406 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:59:19.224373    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-8g97n"]
Jul 13 18:59:19.344822 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:59:19.344779    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87a48a7a-6d63-43f0-8bab-7e29c1a6979a-istio-envoy\") pod \"openshift-ai-inference-openshift-default-655f7c58b4-8g97n\" (UID: \"87a48a7a-6d63-43f0-8bab-7e29c1a6979a\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-8g97n"
Jul 13 18:59:19.345014 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:59:19.344833    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87a48a7a-6d63-43f0-8bab-7e29c1a6979a-credential-socket\") pod \"openshift-ai-inference-openshift-default-655f7c58b4-8g97n\" (UID: \"87a48a7a-6d63-43f0-8bab-7e29c1a6979a\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-8g97n"
Jul 13 18:59:19.345014 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:59:19.344913    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87a48a7a-6d63-43f0-8bab-7e29c1a6979a-istio-token\") pod \"openshift-ai-inference-openshift-default-655f7c58b4-8g97n\" (UID: \"87a48a7a-6d63-43f0-8bab-7e29c1a6979a\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-8g97n"
Jul 13 18:59:19.345014 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:59:19.344949    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87a48a7a-6d63-43f0-8bab-7e29c1a6979a-istio-data\") pod \"openshift-ai-inference-openshift-default-655f7c58b4-8g97n\" (UID: \"87a48a7a-6d63-43f0-8bab-7e29c1a6979a\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-8g97n"
Jul 13 18:59:19.345014 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:59:19.345001    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9z55w\" (UniqueName: \"kubernetes.io/projected/87a48a7a-6d63-43f0-8bab-7e29c1a6979a-kube-api-access-9z55w\") pod \"openshift-ai-inference-openshift-default-655f7c58b4-8g97n\" (UID: \"87a48a7a-6d63-43f0-8bab-7e29c1a6979a\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-8g97n"
Jul 13 18:59:19.345189 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:59:19.345020    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87a48a7a-6d63-43f0-8bab-7e29c1a6979a-workload-certs\") pod \"openshift-ai-inference-openshift-default-655f7c58b4-8g97n\" (UID: \"87a48a7a-6d63-43f0-8bab-7e29c1a6979a\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-8g97n"
Jul 13 18:59:19.345189 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:59:19.345034    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87a48a7a-6d63-43f0-8bab-7e29c1a6979a-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-655f7c58b4-8g97n\" (UID: \"87a48a7a-6d63-43f0-8bab-7e29c1a6979a\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-8g97n"
Jul 13 18:59:19.345189 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:59:19.345071    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87a48a7a-6d63-43f0-8bab-7e29c1a6979a-workload-socket\") pod \"openshift-ai-inference-openshift-default-655f7c58b4-8g97n\" (UID: \"87a48a7a-6d63-43f0-8bab-7e29c1a6979a\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-8g97n"
Jul 13 18:59:19.345189 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:59:19.345089    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87a48a7a-6d63-43f0-8bab-7e29c1a6979a-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-655f7c58b4-8g97n\" (UID: \"87a48a7a-6d63-43f0-8bab-7e29c1a6979a\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-8g97n"
Jul 13 18:59:19.445681 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:59:19.445640    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87a48a7a-6d63-43f0-8bab-7e29c1a6979a-workload-socket\") pod \"openshift-ai-inference-openshift-default-655f7c58b4-8g97n\" (UID: \"87a48a7a-6d63-43f0-8bab-7e29c1a6979a\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-8g97n"
Jul 13 18:59:19.445681 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:59:19.445684    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87a48a7a-6d63-43f0-8bab-7e29c1a6979a-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-655f7c58b4-8g97n\" (UID: \"87a48a7a-6d63-43f0-8bab-7e29c1a6979a\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-8g97n"
Jul 13 18:59:19.445957 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:59:19.445743    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87a48a7a-6d63-43f0-8bab-7e29c1a6979a-istio-envoy\") pod \"openshift-ai-inference-openshift-default-655f7c58b4-8g97n\" (UID: \"87a48a7a-6d63-43f0-8bab-7e29c1a6979a\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-8g97n"
Jul 13 18:59:19.445957 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:59:19.445785    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87a48a7a-6d63-43f0-8bab-7e29c1a6979a-credential-socket\") pod \"openshift-ai-inference-openshift-default-655f7c58b4-8g97n\" (UID: \"87a48a7a-6d63-43f0-8bab-7e29c1a6979a\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-8g97n"
Jul 13 18:59:19.445957 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:59:19.445839    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87a48a7a-6d63-43f0-8bab-7e29c1a6979a-istio-token\") pod \"openshift-ai-inference-openshift-default-655f7c58b4-8g97n\" (UID: \"87a48a7a-6d63-43f0-8bab-7e29c1a6979a\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-8g97n"
Jul 13 18:59:19.445957 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:59:19.445868    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87a48a7a-6d63-43f0-8bab-7e29c1a6979a-istio-data\") pod \"openshift-ai-inference-openshift-default-655f7c58b4-8g97n\" (UID: \"87a48a7a-6d63-43f0-8bab-7e29c1a6979a\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-8g97n"
Jul 13 18:59:19.445957 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:59:19.445925    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9z55w\" (UniqueName: \"kubernetes.io/projected/87a48a7a-6d63-43f0-8bab-7e29c1a6979a-kube-api-access-9z55w\") pod \"openshift-ai-inference-openshift-default-655f7c58b4-8g97n\" (UID: \"87a48a7a-6d63-43f0-8bab-7e29c1a6979a\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-8g97n"
Jul 13 18:59:19.445957 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:59:19.445954    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87a48a7a-6d63-43f0-8bab-7e29c1a6979a-workload-certs\") pod \"openshift-ai-inference-openshift-default-655f7c58b4-8g97n\" (UID: \"87a48a7a-6d63-43f0-8bab-7e29c1a6979a\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-8g97n"
Jul 13 18:59:19.446281 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:59:19.445977    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87a48a7a-6d63-43f0-8bab-7e29c1a6979a-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-655f7c58b4-8g97n\" (UID: \"87a48a7a-6d63-43f0-8bab-7e29c1a6979a\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-8g97n"
Jul 13 18:59:19.446281 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:59:19.446148    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87a48a7a-6d63-43f0-8bab-7e29c1a6979a-workload-socket\") pod \"openshift-ai-inference-openshift-default-655f7c58b4-8g97n\" (UID: \"87a48a7a-6d63-43f0-8bab-7e29c1a6979a\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-8g97n"
Jul 13 18:59:19.446281 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:59:19.446239    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87a48a7a-6d63-43f0-8bab-7e29c1a6979a-credential-socket\") pod \"openshift-ai-inference-openshift-default-655f7c58b4-8g97n\" (UID: \"87a48a7a-6d63-43f0-8bab-7e29c1a6979a\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-8g97n"
Jul 13 18:59:19.446444 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:59:19.446309    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87a48a7a-6d63-43f0-8bab-7e29c1a6979a-istio-data\") pod \"openshift-ai-inference-openshift-default-655f7c58b4-8g97n\" (UID: \"87a48a7a-6d63-43f0-8bab-7e29c1a6979a\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-8g97n"
Jul 13 18:59:19.446658 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:59:19.446449    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87a48a7a-6d63-43f0-8bab-7e29c1a6979a-workload-certs\") pod \"openshift-ai-inference-openshift-default-655f7c58b4-8g97n\" (UID: \"87a48a7a-6d63-43f0-8bab-7e29c1a6979a\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-8g97n"
Jul 13 18:59:19.446658 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:59:19.446525    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87a48a7a-6d63-43f0-8bab-7e29c1a6979a-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-655f7c58b4-8g97n\" (UID: \"87a48a7a-6d63-43f0-8bab-7e29c1a6979a\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-8g97n"
Jul 13 18:59:19.448217 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:59:19.448195    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87a48a7a-6d63-43f0-8bab-7e29c1a6979a-istio-envoy\") pod \"openshift-ai-inference-openshift-default-655f7c58b4-8g97n\" (UID: \"87a48a7a-6d63-43f0-8bab-7e29c1a6979a\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-8g97n"
Jul 13 18:59:19.448608 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:59:19.448587    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87a48a7a-6d63-43f0-8bab-7e29c1a6979a-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-655f7c58b4-8g97n\" (UID: \"87a48a7a-6d63-43f0-8bab-7e29c1a6979a\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-8g97n"
Jul 13 18:59:19.456058 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:59:19.455999    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87a48a7a-6d63-43f0-8bab-7e29c1a6979a-istio-token\") pod \"openshift-ai-inference-openshift-default-655f7c58b4-8g97n\" (UID: \"87a48a7a-6d63-43f0-8bab-7e29c1a6979a\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-8g97n"
Jul 13 18:59:19.456304 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:59:19.456281    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9z55w\" (UniqueName: \"kubernetes.io/projected/87a48a7a-6d63-43f0-8bab-7e29c1a6979a-kube-api-access-9z55w\") pod \"openshift-ai-inference-openshift-default-655f7c58b4-8g97n\" (UID: \"87a48a7a-6d63-43f0-8bab-7e29c1a6979a\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-8g97n"
Jul 13 18:59:19.522773 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:59:19.522729    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-8g97n"
Jul 13 18:59:19.672956 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:59:19.672927    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-8g97n"]
Jul 13 18:59:19.675208 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:59:19.675176    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87a48a7a_6d63_43f0_8bab_7e29c1a6979a.slice/crio-297afd33117d93757f7f08a463e86559774eaca4a31fce9799cb2925fb84a47b WatchSource:0}: Error finding container 297afd33117d93757f7f08a463e86559774eaca4a31fce9799cb2925fb84a47b: Status 404 returned error can't find the container with id 297afd33117d93757f7f08a463e86559774eaca4a31fce9799cb2925fb84a47b
Jul 13 18:59:20.564905 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:59:20.564865    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-8g97n" event={"ID":"87a48a7a-6d63-43f0-8bab-7e29c1a6979a","Type":"ContainerStarted","Data":"297afd33117d93757f7f08a463e86559774eaca4a31fce9799cb2925fb84a47b"}
Jul 13 18:59:23.857535 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:59:23.857490    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 13 18:59:23.857906 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:59:23.857584    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 13 18:59:23.857906 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:59:23.857637    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 13 18:59:24.592292 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:59:24.592248    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-8g97n" event={"ID":"87a48a7a-6d63-43f0-8bab-7e29c1a6979a","Type":"ContainerStarted","Data":"dd3948ff3e8e5c8ecb3d67eb4faf9886831fcaafebef990c0de293d4aa95f089"}
Jul 13 18:59:24.615175 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:59:24.615119    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-8g97n" podStartSLOduration=1.435128141 podStartE2EDuration="5.615101054s" totalImagesPullingTime="4.179972913s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 18:59:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:59:24.613069815 +0000 UTC m=+551.138203245" watchObservedRunningTime="2026-07-13 18:59:24.615101054 +0000 UTC m=+551.140234471"
Jul 13 18:59:25.523491 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:59:25.523454    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-8g97n"
Jul 13 18:59:25.528389 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:59:25.528362    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-8g97n"
Jul 13 18:59:25.596835 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:59:25.596799    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-8g97n"
Jul 13 18:59:25.597895 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:59:25.597872    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-8g97n"
Jul 13 18:59:26.213513 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:59:26.213476    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-5ff584b858-xkf9w"]
Jul 13 18:59:26.216250 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:59:26.216234    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5ff584b858-xkf9w"
Jul 13 18:59:26.219631 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:59:26.219610    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Jul 13 18:59:26.219791 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:59:26.219609    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Jul 13 18:59:26.219851 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:59:26.219837    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Jul 13 18:59:26.221282 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:59:26.221267    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-r6lvn\""
Jul 13 18:59:26.221823 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:59:26.221810    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Jul 13 18:59:26.222064 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:59:26.222045    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Jul 13 18:59:26.241887 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:59:26.241851    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5ff584b858-xkf9w"]
Jul 13 18:59:26.306062 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:59:26.306025    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/18724f8f-9798-46b5-a6a1-574ab45fcb20-metrics-cert\") pod \"lws-controller-manager-5ff584b858-xkf9w\" (UID: \"18724f8f-9798-46b5-a6a1-574ab45fcb20\") " pod="openshift-lws-operator/lws-controller-manager-5ff584b858-xkf9w"
Jul 13 18:59:26.306240 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:59:26.306075    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/18724f8f-9798-46b5-a6a1-574ab45fcb20-manager-config\") pod \"lws-controller-manager-5ff584b858-xkf9w\" (UID: \"18724f8f-9798-46b5-a6a1-574ab45fcb20\") " pod="openshift-lws-operator/lws-controller-manager-5ff584b858-xkf9w"
Jul 13 18:59:26.306240 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:59:26.306132    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4grv7\" (UniqueName: \"kubernetes.io/projected/18724f8f-9798-46b5-a6a1-574ab45fcb20-kube-api-access-4grv7\") pod \"lws-controller-manager-5ff584b858-xkf9w\" (UID: \"18724f8f-9798-46b5-a6a1-574ab45fcb20\") " pod="openshift-lws-operator/lws-controller-manager-5ff584b858-xkf9w"
Jul 13 18:59:26.306240 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:59:26.306156    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/18724f8f-9798-46b5-a6a1-574ab45fcb20-cert\") pod \"lws-controller-manager-5ff584b858-xkf9w\" (UID: \"18724f8f-9798-46b5-a6a1-574ab45fcb20\") " pod="openshift-lws-operator/lws-controller-manager-5ff584b858-xkf9w"
Jul 13 18:59:26.407452 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:59:26.407409    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/18724f8f-9798-46b5-a6a1-574ab45fcb20-manager-config\") pod \"lws-controller-manager-5ff584b858-xkf9w\" (UID: \"18724f8f-9798-46b5-a6a1-574ab45fcb20\") " pod="openshift-lws-operator/lws-controller-manager-5ff584b858-xkf9w"
Jul 13 18:59:26.407650 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:59:26.407503    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4grv7\" (UniqueName: \"kubernetes.io/projected/18724f8f-9798-46b5-a6a1-574ab45fcb20-kube-api-access-4grv7\") pod \"lws-controller-manager-5ff584b858-xkf9w\" (UID: \"18724f8f-9798-46b5-a6a1-574ab45fcb20\") " pod="openshift-lws-operator/lws-controller-manager-5ff584b858-xkf9w"
Jul 13 18:59:26.407650 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:59:26.407528    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/18724f8f-9798-46b5-a6a1-574ab45fcb20-cert\") pod \"lws-controller-manager-5ff584b858-xkf9w\" (UID: \"18724f8f-9798-46b5-a6a1-574ab45fcb20\") " pod="openshift-lws-operator/lws-controller-manager-5ff584b858-xkf9w"
Jul 13 18:59:26.407650 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:59:26.407569    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/18724f8f-9798-46b5-a6a1-574ab45fcb20-metrics-cert\") pod \"lws-controller-manager-5ff584b858-xkf9w\" (UID: \"18724f8f-9798-46b5-a6a1-574ab45fcb20\") " pod="openshift-lws-operator/lws-controller-manager-5ff584b858-xkf9w"
Jul 13 18:59:26.408124 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:59:26.408101    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/18724f8f-9798-46b5-a6a1-574ab45fcb20-manager-config\") pod \"lws-controller-manager-5ff584b858-xkf9w\" (UID: \"18724f8f-9798-46b5-a6a1-574ab45fcb20\") " pod="openshift-lws-operator/lws-controller-manager-5ff584b858-xkf9w"
Jul 13 18:59:26.410344 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:59:26.410316    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/18724f8f-9798-46b5-a6a1-574ab45fcb20-cert\") pod \"lws-controller-manager-5ff584b858-xkf9w\" (UID: \"18724f8f-9798-46b5-a6a1-574ab45fcb20\") " pod="openshift-lws-operator/lws-controller-manager-5ff584b858-xkf9w"
Jul 13 18:59:26.410468 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:59:26.410323    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/18724f8f-9798-46b5-a6a1-574ab45fcb20-metrics-cert\") pod \"lws-controller-manager-5ff584b858-xkf9w\" (UID: \"18724f8f-9798-46b5-a6a1-574ab45fcb20\") " pod="openshift-lws-operator/lws-controller-manager-5ff584b858-xkf9w"
Jul 13 18:59:26.417125 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:59:26.417096    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4grv7\" (UniqueName: \"kubernetes.io/projected/18724f8f-9798-46b5-a6a1-574ab45fcb20-kube-api-access-4grv7\") pod \"lws-controller-manager-5ff584b858-xkf9w\" (UID: \"18724f8f-9798-46b5-a6a1-574ab45fcb20\") " pod="openshift-lws-operator/lws-controller-manager-5ff584b858-xkf9w"
Jul 13 18:59:26.526210 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:59:26.526180    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5ff584b858-xkf9w"
Jul 13 18:59:26.674998 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:59:26.674970    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5ff584b858-xkf9w"]
Jul 13 18:59:26.677621 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:59:26.677588    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18724f8f_9798_46b5_a6a1_574ab45fcb20.slice/crio-1809743f46ef08d1198c3bf7e82124b18d0c1848b3d8cef51018d10dfe6fcd62 WatchSource:0}: Error finding container 1809743f46ef08d1198c3bf7e82124b18d0c1848b3d8cef51018d10dfe6fcd62: Status 404 returned error can't find the container with id 1809743f46ef08d1198c3bf7e82124b18d0c1848b3d8cef51018d10dfe6fcd62
Jul 13 18:59:27.607923 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:59:27.607883    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5ff584b858-xkf9w" event={"ID":"18724f8f-9798-46b5-a6a1-574ab45fcb20","Type":"ContainerStarted","Data":"1809743f46ef08d1198c3bf7e82124b18d0c1848b3d8cef51018d10dfe6fcd62"}
Jul 13 18:59:30.624298 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:59:30.624256    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5ff584b858-xkf9w" event={"ID":"18724f8f-9798-46b5-a6a1-574ab45fcb20","Type":"ContainerStarted","Data":"e817ef85d53cc610273008cbc14794e26b0450cd06dce902912f0aa35f1c6144"}
Jul 13 18:59:30.624693 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:59:30.624400    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-5ff584b858-xkf9w"
Jul 13 18:59:30.642823 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:59:30.642769    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-5ff584b858-xkf9w" podStartSLOduration=1.2376420750000001 podStartE2EDuration="4.642753242s" totalImagesPullingTime="3.405111167s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 18:59:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 18:59:30.640736886 +0000 UTC m=+557.165870302" watchObservedRunningTime="2026-07-13 18:59:30.642753242 +0000 UTC m=+557.167886658"
Jul 13 18:59:41.631457 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:59:41.631419    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-5ff584b858-xkf9w"
Jul 13 18:59:55.373636 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:59:55.373553    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-4tc92"]
Jul 13 18:59:55.379576 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:59:55.379555    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-4tc92"
Jul 13 18:59:55.382438 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:59:55.382411    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Jul 13 18:59:55.382663 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:59:55.382648    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 13 18:59:55.383366 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:59:55.383344    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Jul 13 18:59:55.383548 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:59:55.383384    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 13 18:59:55.383816 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:59:55.383394    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-h74mc\""
Jul 13 18:59:55.386627 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:59:55.386599    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-4tc92"]
Jul 13 18:59:55.465799 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:59:55.465757    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbgm5\" (UniqueName: \"kubernetes.io/projected/6b37279a-c1b7-4e01-b19a-b6b69bf6be78-kube-api-access-xbgm5\") pod \"kuadrant-console-plugin-85fd4b8b9d-4tc92\" (UID: \"6b37279a-c1b7-4e01-b19a-b6b69bf6be78\") " pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-4tc92"
Jul 13 18:59:55.466009 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:59:55.465920    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/6b37279a-c1b7-4e01-b19a-b6b69bf6be78-plugin-serving-cert\") pod \"kuadrant-console-plugin-85fd4b8b9d-4tc92\" (UID: \"6b37279a-c1b7-4e01-b19a-b6b69bf6be78\") " pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-4tc92"
Jul 13 18:59:55.466009 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:59:55.465977    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/6b37279a-c1b7-4e01-b19a-b6b69bf6be78-nginx-conf\") pod \"kuadrant-console-plugin-85fd4b8b9d-4tc92\" (UID: \"6b37279a-c1b7-4e01-b19a-b6b69bf6be78\") " pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-4tc92"
Jul 13 18:59:55.567083 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:59:55.567032    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/6b37279a-c1b7-4e01-b19a-b6b69bf6be78-plugin-serving-cert\") pod \"kuadrant-console-plugin-85fd4b8b9d-4tc92\" (UID: \"6b37279a-c1b7-4e01-b19a-b6b69bf6be78\") " pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-4tc92"
Jul 13 18:59:55.567293 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:59:55.567098    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/6b37279a-c1b7-4e01-b19a-b6b69bf6be78-nginx-conf\") pod \"kuadrant-console-plugin-85fd4b8b9d-4tc92\" (UID: \"6b37279a-c1b7-4e01-b19a-b6b69bf6be78\") " pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-4tc92"
Jul 13 18:59:55.567293 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:59:55.567139    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbgm5\" (UniqueName: \"kubernetes.io/projected/6b37279a-c1b7-4e01-b19a-b6b69bf6be78-kube-api-access-xbgm5\") pod \"kuadrant-console-plugin-85fd4b8b9d-4tc92\" (UID: \"6b37279a-c1b7-4e01-b19a-b6b69bf6be78\") " pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-4tc92"
Jul 13 18:59:55.567293 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:59:55.567208    2565 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found
Jul 13 18:59:55.567463 ip-10-0-128-215 kubenswrapper[2565]: E0713 18:59:55.567300    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b37279a-c1b7-4e01-b19a-b6b69bf6be78-plugin-serving-cert podName:6b37279a-c1b7-4e01-b19a-b6b69bf6be78 nodeName:}" failed. No retries permitted until 2026-07-13 18:59:56.06727822 +0000 UTC m=+582.592411614 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/6b37279a-c1b7-4e01-b19a-b6b69bf6be78-plugin-serving-cert") pod "kuadrant-console-plugin-85fd4b8b9d-4tc92" (UID: "6b37279a-c1b7-4e01-b19a-b6b69bf6be78") : secret "plugin-serving-cert" not found
Jul 13 18:59:55.567830 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:59:55.567810    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/6b37279a-c1b7-4e01-b19a-b6b69bf6be78-nginx-conf\") pod \"kuadrant-console-plugin-85fd4b8b9d-4tc92\" (UID: \"6b37279a-c1b7-4e01-b19a-b6b69bf6be78\") " pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-4tc92"
Jul 13 18:59:55.588557 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:59:55.588518    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbgm5\" (UniqueName: \"kubernetes.io/projected/6b37279a-c1b7-4e01-b19a-b6b69bf6be78-kube-api-access-xbgm5\") pod \"kuadrant-console-plugin-85fd4b8b9d-4tc92\" (UID: \"6b37279a-c1b7-4e01-b19a-b6b69bf6be78\") " pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-4tc92"
Jul 13 18:59:56.072204 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:59:56.072163    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/6b37279a-c1b7-4e01-b19a-b6b69bf6be78-plugin-serving-cert\") pod \"kuadrant-console-plugin-85fd4b8b9d-4tc92\" (UID: \"6b37279a-c1b7-4e01-b19a-b6b69bf6be78\") " pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-4tc92"
Jul 13 18:59:56.074735 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:59:56.074693    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/6b37279a-c1b7-4e01-b19a-b6b69bf6be78-plugin-serving-cert\") pod \"kuadrant-console-plugin-85fd4b8b9d-4tc92\" (UID: \"6b37279a-c1b7-4e01-b19a-b6b69bf6be78\") " pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-4tc92"
Jul 13 18:59:56.290999 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:59:56.290952    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-4tc92"
Jul 13 18:59:56.434882 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:59:56.434806    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-4tc92"]
Jul 13 18:59:56.437166 ip-10-0-128-215 kubenswrapper[2565]: W0713 18:59:56.437134    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6b37279a_c1b7_4e01_b19a_b6b69bf6be78.slice/crio-cecf46862a73ecb9bcbee7aae976deca18e6f356b542265680323322bd7231eb WatchSource:0}: Error finding container cecf46862a73ecb9bcbee7aae976deca18e6f356b542265680323322bd7231eb: Status 404 returned error can't find the container with id cecf46862a73ecb9bcbee7aae976deca18e6f356b542265680323322bd7231eb
Jul 13 18:59:56.764679 ip-10-0-128-215 kubenswrapper[2565]: I0713 18:59:56.764642    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-4tc92" event={"ID":"6b37279a-c1b7-4e01-b19a-b6b69bf6be78","Type":"ContainerStarted","Data":"cecf46862a73ecb9bcbee7aae976deca18e6f356b542265680323322bd7231eb"}
Jul 13 19:00:01.793922 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:00:01.793881    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-4tc92" event={"ID":"6b37279a-c1b7-4e01-b19a-b6b69bf6be78","Type":"ContainerStarted","Data":"4255b22bc0399525b352754581a5151429d4a0ade83a700a30317a94f3cadc36"}
Jul 13 19:00:01.811323 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:00:01.811271    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-4tc92" podStartSLOduration=2.182446136 podStartE2EDuration="6.811248416s" totalImagesPullingTime="4.62880228s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 18:59:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 19:00:01.810173092 +0000 UTC m=+588.335306505" watchObservedRunningTime="2026-07-13 19:00:01.811248416 +0000 UTC m=+588.336381832"
Jul 13 19:00:13.881324 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:00:13.881291    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2bscc_2f0655bd-4a09-4314-8c1b-e30191eaec0e/ovn-acl-logging/0.log"
Jul 13 19:00:13.885088 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:00:13.885061    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2bscc_2f0655bd-4a09-4314-8c1b-e30191eaec0e/ovn-acl-logging/0.log"
Jul 13 19:00:38.803087 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:00:38.802994    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-hnvp6"]
Jul 13 19:00:38.806415 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:00:38.806390    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-69574b596d-hnvp6"
Jul 13 19:00:38.809077 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:00:38.809047    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Jul 13 19:00:38.817174 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:00:38.817143    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-hnvp6"]
Jul 13 19:00:38.844649 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:00:38.844610    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-hnvp6"]
Jul 13 19:00:38.856069 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:00:38.856031    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jpcs\" (UniqueName: \"kubernetes.io/projected/bee0c256-af28-4777-89d9-411b937b62fc-kube-api-access-7jpcs\") pod \"limitador-limitador-69574b596d-hnvp6\" (UID: \"bee0c256-af28-4777-89d9-411b937b62fc\") " pod="kuadrant-system/limitador-limitador-69574b596d-hnvp6"
Jul 13 19:00:38.856215 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:00:38.856096    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/bee0c256-af28-4777-89d9-411b937b62fc-config-file\") pod \"limitador-limitador-69574b596d-hnvp6\" (UID: \"bee0c256-af28-4777-89d9-411b937b62fc\") " pod="kuadrant-system/limitador-limitador-69574b596d-hnvp6"
Jul 13 19:00:38.956598 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:00:38.956556    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jpcs\" (UniqueName: \"kubernetes.io/projected/bee0c256-af28-4777-89d9-411b937b62fc-kube-api-access-7jpcs\") pod \"limitador-limitador-69574b596d-hnvp6\" (UID: \"bee0c256-af28-4777-89d9-411b937b62fc\") " pod="kuadrant-system/limitador-limitador-69574b596d-hnvp6"
Jul 13 19:00:38.956824 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:00:38.956630    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/bee0c256-af28-4777-89d9-411b937b62fc-config-file\") pod \"limitador-limitador-69574b596d-hnvp6\" (UID: \"bee0c256-af28-4777-89d9-411b937b62fc\") " pod="kuadrant-system/limitador-limitador-69574b596d-hnvp6"
Jul 13 19:00:38.957348 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:00:38.957327    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/bee0c256-af28-4777-89d9-411b937b62fc-config-file\") pod \"limitador-limitador-69574b596d-hnvp6\" (UID: \"bee0c256-af28-4777-89d9-411b937b62fc\") " pod="kuadrant-system/limitador-limitador-69574b596d-hnvp6"
Jul 13 19:00:38.966406 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:00:38.966372    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jpcs\" (UniqueName: \"kubernetes.io/projected/bee0c256-af28-4777-89d9-411b937b62fc-kube-api-access-7jpcs\") pod \"limitador-limitador-69574b596d-hnvp6\" (UID: \"bee0c256-af28-4777-89d9-411b937b62fc\") " pod="kuadrant-system/limitador-limitador-69574b596d-hnvp6"
Jul 13 19:00:39.117994 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:00:39.117903    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-69574b596d-hnvp6"
Jul 13 19:00:39.264177 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:00:39.264146    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-69574b596d-hnvp6"]
Jul 13 19:00:39.266907 ip-10-0-128-215 kubenswrapper[2565]: W0713 19:00:39.266879    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbee0c256_af28_4777_89d9_411b937b62fc.slice/crio-4ae2d6ad0e543a48ea656f7d108ef2fa7106b328efa4dc6fbbf340b824e82583 WatchSource:0}: Error finding container 4ae2d6ad0e543a48ea656f7d108ef2fa7106b328efa4dc6fbbf340b824e82583: Status 404 returned error can't find the container with id 4ae2d6ad0e543a48ea656f7d108ef2fa7106b328efa4dc6fbbf340b824e82583
Jul 13 19:00:39.981871 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:00:39.981833    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-69574b596d-hnvp6" event={"ID":"bee0c256-af28-4777-89d9-411b937b62fc","Type":"ContainerStarted","Data":"4ae2d6ad0e543a48ea656f7d108ef2fa7106b328efa4dc6fbbf340b824e82583"}
Jul 13 19:01:40.511530 ip-10-0-128-215 kubenswrapper[2565]: E0713 19:01:40.511483    2565 log.go:32] "PullImage from image service failed" err="rpc error: code = Unknown desc = unable to pull image or OCI artifact: pull image err: copying system image from manifest list: reading signatures: reading signature from https://registry.redhat.io/containers/sigstore/rhcl-1/limitador-rhel9@sha256=e40f1438524bd554d9e23b0ddbaaa4ccf42d91ee5912ebdb372d11f734b9b524/signature-5: received unexpected HTTP status: 500 Internal Server Error; artifact err: provided artifact is a container image" image="registry.redhat.io/rhcl-1/limitador-rhel9@sha256:0b02c000d9a175235223ec5916345a1ccedeedc25c87cc0638e8554e6c318474"
Jul 13 19:01:40.511962 ip-10-0-128-215 kubenswrapper[2565]: E0713 19:01:40.511611    2565 kuberuntime_manager.go:1459] "Unhandled Error" err="container limitador start failed in pod limitador-limitador-69574b596d-hnvp6_kuadrant-system(bee0c256-af28-4777-89d9-411b937b62fc): ErrImagePull: unable to pull image or OCI artifact: pull image err: copying system image from manifest list: reading signatures: reading signature from https://registry.redhat.io/containers/sigstore/rhcl-1/limitador-rhel9@sha256=e40f1438524bd554d9e23b0ddbaaa4ccf42d91ee5912ebdb372d11f734b9b524/signature-5: received unexpected HTTP status: 500 Internal Server Error; artifact err: provided artifact is a container image" logger="UnhandledError"
Jul 13 19:01:40.511962 ip-10-0-128-215 kubenswrapper[2565]: E0713 19:01:40.511646    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"limitador\" with ErrImagePull: \"unable to pull image or OCI artifact: pull image err: copying system image from manifest list: reading signatures: reading signature from https://registry.redhat.io/containers/sigstore/rhcl-1/limitador-rhel9@sha256=e40f1438524bd554d9e23b0ddbaaa4ccf42d91ee5912ebdb372d11f734b9b524/signature-5: received unexpected HTTP status: 500 Internal Server Error; artifact err: provided artifact is a container image\"" pod="kuadrant-system/limitador-limitador-69574b596d-hnvp6" podUID="bee0c256-af28-4777-89d9-411b937b62fc"
Jul 13 19:01:41.284798 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:01:41.284777    2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 13 19:01:41.285008 ip-10-0-128-215 kubenswrapper[2565]: E0713 19:01:41.284983    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"limitador\" with ImagePullBackOff: \"Back-off pulling image \\\"registry.redhat.io/rhcl-1/limitador-rhel9@sha256:0b02c000d9a175235223ec5916345a1ccedeedc25c87cc0638e8554e6c318474\\\": ErrImagePull: unable to pull image or OCI artifact: pull image err: copying system image from manifest list: reading signatures: reading signature from https://registry.redhat.io/containers/sigstore/rhcl-1/limitador-rhel9@sha256=e40f1438524bd554d9e23b0ddbaaa4ccf42d91ee5912ebdb372d11f734b9b524/signature-5: received unexpected HTTP status: 500 Internal Server Error; artifact err: provided artifact is a container image\"" pod="kuadrant-system/limitador-limitador-69574b596d-hnvp6" podUID="bee0c256-af28-4777-89d9-411b937b62fc"
Jul 13 19:02:01.392177 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:02:01.392135    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-69574b596d-hnvp6" event={"ID":"bee0c256-af28-4777-89d9-411b937b62fc","Type":"ContainerStarted","Data":"131585475d5d82f9b427cc8a373654d4b15c86af5888c80865b2db7727d069c9"}
Jul 13 19:02:01.392583 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:02:01.392359    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-69574b596d-hnvp6"
Jul 13 19:02:01.414846 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:02:01.414781    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-69574b596d-hnvp6" podStartSLOduration=2.229342828 podStartE2EDuration="1m23.414762754s" totalImagesPullingTime="1m21.185419926s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 19:00:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=1 observedRunningTime="2026-07-13 19:02:01.411316522 +0000 UTC m=+707.936449937" watchObservedRunningTime="2026-07-13 19:02:01.414762754 +0000 UTC m=+707.939896170"
Jul 13 19:02:12.397881 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:02:12.397794    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-69574b596d-hnvp6"
Jul 13 19:02:28.289814 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:02:28.289773    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-6b69b6d547-f58tf"]
Jul 13 19:02:28.290251 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:02:28.290012    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve/kserve-controller-manager-6b69b6d547-f58tf" podUID="b1721a10-9182-4f1e-9146-f546ca94fed6" containerName="manager" containerID="cri-o://8432ca785e1647d1c706aa41386cd59354b8fada83f39bc4af588198ded502c4" gracePeriod=10
Jul 13 19:02:28.531305 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:02:28.531269    2565 generic.go:356] "Generic (PLEG): container finished" podID="b1721a10-9182-4f1e-9146-f546ca94fed6" containerID="8432ca785e1647d1c706aa41386cd59354b8fada83f39bc4af588198ded502c4" exitCode=0
Jul 13 19:02:28.531470 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:02:28.531320    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-6b69b6d547-f58tf" event={"ID":"b1721a10-9182-4f1e-9146-f546ca94fed6","Type":"ContainerDied","Data":"8432ca785e1647d1c706aa41386cd59354b8fada83f39bc4af588198ded502c4"}
Jul 13 19:02:28.550533 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:02:28.550459    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-6b69b6d547-f58tf"
Jul 13 19:02:28.679727 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:02:28.679662    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hlcz9\" (UniqueName: \"kubernetes.io/projected/b1721a10-9182-4f1e-9146-f546ca94fed6-kube-api-access-hlcz9\") pod \"b1721a10-9182-4f1e-9146-f546ca94fed6\" (UID: \"b1721a10-9182-4f1e-9146-f546ca94fed6\") "
Jul 13 19:02:28.679901 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:02:28.679747    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b1721a10-9182-4f1e-9146-f546ca94fed6-cert\") pod \"b1721a10-9182-4f1e-9146-f546ca94fed6\" (UID: \"b1721a10-9182-4f1e-9146-f546ca94fed6\") "
Jul 13 19:02:28.682296 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:02:28.682264    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1721a10-9182-4f1e-9146-f546ca94fed6-cert" (OuterVolumeSpecName: "cert") pod "b1721a10-9182-4f1e-9146-f546ca94fed6" (UID: "b1721a10-9182-4f1e-9146-f546ca94fed6"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 19:02:28.682426 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:02:28.682264    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1721a10-9182-4f1e-9146-f546ca94fed6-kube-api-access-hlcz9" (OuterVolumeSpecName: "kube-api-access-hlcz9") pod "b1721a10-9182-4f1e-9146-f546ca94fed6" (UID: "b1721a10-9182-4f1e-9146-f546ca94fed6"). InnerVolumeSpecName "kube-api-access-hlcz9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 19:02:28.781040 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:02:28.780992    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hlcz9\" (UniqueName: \"kubernetes.io/projected/b1721a10-9182-4f1e-9146-f546ca94fed6-kube-api-access-hlcz9\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 13 19:02:28.781040 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:02:28.781028    2565 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b1721a10-9182-4f1e-9146-f546ca94fed6-cert\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 13 19:02:29.537436 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:02:29.537400    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-6b69b6d547-f58tf"
Jul 13 19:02:29.537888 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:02:29.537447    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-6b69b6d547-f58tf" event={"ID":"b1721a10-9182-4f1e-9146-f546ca94fed6","Type":"ContainerDied","Data":"14e9eadcccb0ac65a952a071cdc5985f79d591a9422f0b70d85d5fbc378e0344"}
Jul 13 19:02:29.537888 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:02:29.537512    2565 scope.go:117] "RemoveContainer" containerID="8432ca785e1647d1c706aa41386cd59354b8fada83f39bc4af588198ded502c4"
Jul 13 19:02:29.564052 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:02:29.564007    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-6b69b6d547-f58tf"]
Jul 13 19:02:29.568268 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:02:29.568238    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/kserve-controller-manager-6b69b6d547-f58tf"]
Jul 13 19:02:30.048281 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:02:30.048237    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b1721a10-9182-4f1e-9146-f546ca94fed6" path="/var/lib/kubelet/pods/b1721a10-9182-4f1e-9146-f546ca94fed6/volumes"
Jul 13 19:03:20.658820 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:03:20.658779    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-ctct5"]
Jul 13 19:03:20.659280 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:03:20.659186    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1721a10-9182-4f1e-9146-f546ca94fed6" containerName="manager"
Jul 13 19:03:20.659280 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:03:20.659202    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1721a10-9182-4f1e-9146-f546ca94fed6" containerName="manager"
Jul 13 19:03:20.659280 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:03:20.659281    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="b1721a10-9182-4f1e-9146-f546ca94fed6" containerName="manager"
Jul 13 19:03:20.661956 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:03:20.661936    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-ctct5"
Jul 13 19:03:20.674373 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:03:20.674341    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-ctct5"]
Jul 13 19:03:20.845515 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:03:20.845477    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60a3ab50-3f54-418e-8b39-4bbd8b5d92bf-model-cache\") pod \"s3-init-ctct5\" (UID: \"60a3ab50-3f54-418e-8b39-4bbd8b5d92bf\") " pod="kserve/s3-init-ctct5"
Jul 13 19:03:20.845759 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:03:20.845544    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlhxf\" (UniqueName: \"kubernetes.io/projected/60a3ab50-3f54-418e-8b39-4bbd8b5d92bf-kube-api-access-dlhxf\") pod \"s3-init-ctct5\" (UID: \"60a3ab50-3f54-418e-8b39-4bbd8b5d92bf\") " pod="kserve/s3-init-ctct5"
Jul 13 19:03:20.946694 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:03:20.946593    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlhxf\" (UniqueName: \"kubernetes.io/projected/60a3ab50-3f54-418e-8b39-4bbd8b5d92bf-kube-api-access-dlhxf\") pod \"s3-init-ctct5\" (UID: \"60a3ab50-3f54-418e-8b39-4bbd8b5d92bf\") " pod="kserve/s3-init-ctct5"
Jul 13 19:03:20.946892 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:03:20.946729    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60a3ab50-3f54-418e-8b39-4bbd8b5d92bf-model-cache\") pod \"s3-init-ctct5\" (UID: \"60a3ab50-3f54-418e-8b39-4bbd8b5d92bf\") " pod="kserve/s3-init-ctct5"
Jul 13 19:03:20.947087 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:03:20.947070    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60a3ab50-3f54-418e-8b39-4bbd8b5d92bf-model-cache\") pod \"s3-init-ctct5\" (UID: \"60a3ab50-3f54-418e-8b39-4bbd8b5d92bf\") " pod="kserve/s3-init-ctct5"
Jul 13 19:03:20.957295 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:03:20.957257    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlhxf\" (UniqueName: \"kubernetes.io/projected/60a3ab50-3f54-418e-8b39-4bbd8b5d92bf-kube-api-access-dlhxf\") pod \"s3-init-ctct5\" (UID: \"60a3ab50-3f54-418e-8b39-4bbd8b5d92bf\") " pod="kserve/s3-init-ctct5"
Jul 13 19:03:20.971159 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:03:20.971125    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-ctct5"
Jul 13 19:03:21.117588 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:03:21.117421    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-ctct5"]
Jul 13 19:03:21.120741 ip-10-0-128-215 kubenswrapper[2565]: W0713 19:03:21.120684    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod60a3ab50_3f54_418e_8b39_4bbd8b5d92bf.slice/crio-e07a321299b9420cce9171ea67f2d1c505e4c848a99b24d240f342bfee854599 WatchSource:0}: Error finding container e07a321299b9420cce9171ea67f2d1c505e4c848a99b24d240f342bfee854599: Status 404 returned error can't find the container with id e07a321299b9420cce9171ea67f2d1c505e4c848a99b24d240f342bfee854599
Jul 13 19:03:21.803247 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:03:21.803212    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-ctct5" event={"ID":"60a3ab50-3f54-418e-8b39-4bbd8b5d92bf","Type":"ContainerStarted","Data":"e07a321299b9420cce9171ea67f2d1c505e4c848a99b24d240f342bfee854599"}
Jul 13 19:03:25.828809 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:03:25.828764    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-ctct5" event={"ID":"60a3ab50-3f54-418e-8b39-4bbd8b5d92bf","Type":"ContainerStarted","Data":"82acdf945ec02169427b9d032c68fb10d3a9db9fe1cec28157370426cf35add1"}
Jul 13 19:03:28.844461 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:03:28.844424    2565 generic.go:356] "Generic (PLEG): container finished" podID="60a3ab50-3f54-418e-8b39-4bbd8b5d92bf" containerID="82acdf945ec02169427b9d032c68fb10d3a9db9fe1cec28157370426cf35add1" exitCode=0
Jul 13 19:03:28.844881 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:03:28.844497    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-ctct5" event={"ID":"60a3ab50-3f54-418e-8b39-4bbd8b5d92bf","Type":"ContainerDied","Data":"82acdf945ec02169427b9d032c68fb10d3a9db9fe1cec28157370426cf35add1"}
Jul 13 19:03:34.036857 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:03:34.036831    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Jul 13 19:03:34.879203 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:03:34.879168    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-ctct5" event={"ID":"60a3ab50-3f54-418e-8b39-4bbd8b5d92bf","Type":"ContainerStarted","Data":"4bddf7f9b1bade70745792bcc17504b760d74a09256ad22c4419976a718ac9e9"}
Jul 13 19:03:34.900659 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:03:34.900602    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-init-ctct5" podStartSLOduration=2.943217396 podStartE2EDuration="14.900584677s" totalImagesPullingTime="9.722110451s" totalInitContainerRuntime="2.23525683s" isStatefulPod=false podCreationTimestamp="2026-07-13 19:03:20 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 19:03:34.899302456 +0000 UTC m=+801.424435873" watchObservedRunningTime="2026-07-13 19:03:34.900584677 +0000 UTC m=+801.425718092"
Jul 13 19:04:40.197222 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:04:40.197187    2565 generic.go:356] "Generic (PLEG): container finished" podID="60a3ab50-3f54-418e-8b39-4bbd8b5d92bf" containerID="4bddf7f9b1bade70745792bcc17504b760d74a09256ad22c4419976a718ac9e9" exitCode=0
Jul 13 19:04:40.197662 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:04:40.197262    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-ctct5" event={"ID":"60a3ab50-3f54-418e-8b39-4bbd8b5d92bf","Type":"ContainerDied","Data":"4bddf7f9b1bade70745792bcc17504b760d74a09256ad22c4419976a718ac9e9"}
Jul 13 19:04:41.335502 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:04:41.335477    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-ctct5"
Jul 13 19:04:41.464438 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:04:41.464364    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60a3ab50-3f54-418e-8b39-4bbd8b5d92bf-model-cache\") pod \"60a3ab50-3f54-418e-8b39-4bbd8b5d92bf\" (UID: \"60a3ab50-3f54-418e-8b39-4bbd8b5d92bf\") "
Jul 13 19:04:41.464584 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:04:41.464496    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dlhxf\" (UniqueName: \"kubernetes.io/projected/60a3ab50-3f54-418e-8b39-4bbd8b5d92bf-kube-api-access-dlhxf\") pod \"60a3ab50-3f54-418e-8b39-4bbd8b5d92bf\" (UID: \"60a3ab50-3f54-418e-8b39-4bbd8b5d92bf\") "
Jul 13 19:04:41.466684 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:04:41.466655    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60a3ab50-3f54-418e-8b39-4bbd8b5d92bf-kube-api-access-dlhxf" (OuterVolumeSpecName: "kube-api-access-dlhxf") pod "60a3ab50-3f54-418e-8b39-4bbd8b5d92bf" (UID: "60a3ab50-3f54-418e-8b39-4bbd8b5d92bf"). InnerVolumeSpecName "kube-api-access-dlhxf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 19:04:41.482957 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:04:41.482925    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60a3ab50-3f54-418e-8b39-4bbd8b5d92bf-model-cache" (OuterVolumeSpecName: "model-cache") pod "60a3ab50-3f54-418e-8b39-4bbd8b5d92bf" (UID: "60a3ab50-3f54-418e-8b39-4bbd8b5d92bf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:04:41.565898 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:04:41.565860    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dlhxf\" (UniqueName: \"kubernetes.io/projected/60a3ab50-3f54-418e-8b39-4bbd8b5d92bf-kube-api-access-dlhxf\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 13 19:04:41.565898 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:04:41.565892    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60a3ab50-3f54-418e-8b39-4bbd8b5d92bf-model-cache\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 13 19:04:42.207756 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:04:42.207730    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-ctct5"
Jul 13 19:04:42.207756 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:04:42.207741    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-ctct5" event={"ID":"60a3ab50-3f54-418e-8b39-4bbd8b5d92bf","Type":"ContainerDied","Data":"e07a321299b9420cce9171ea67f2d1c505e4c848a99b24d240f342bfee854599"}
Jul 13 19:04:42.208054 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:04:42.207777    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e07a321299b9420cce9171ea67f2d1c505e4c848a99b24d240f342bfee854599"
Jul 13 19:05:04.754988 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:05:04.754909    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-fqc69"]
Jul 13 19:05:04.755454 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:05:04.755304    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60a3ab50-3f54-418e-8b39-4bbd8b5d92bf" containerName="download-hf-model"
Jul 13 19:05:04.755454 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:05:04.755321    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="60a3ab50-3f54-418e-8b39-4bbd8b5d92bf" containerName="download-hf-model"
Jul 13 19:05:04.755454 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:05:04.755333    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60a3ab50-3f54-418e-8b39-4bbd8b5d92bf" containerName="s3-init"
Jul 13 19:05:04.755454 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:05:04.755339    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="60a3ab50-3f54-418e-8b39-4bbd8b5d92bf" containerName="s3-init"
Jul 13 19:05:04.755454 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:05:04.755422    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="60a3ab50-3f54-418e-8b39-4bbd8b5d92bf" containerName="s3-init"
Jul 13 19:05:04.759874 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:05:04.759854    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-fqc69"
Jul 13 19:05:04.762774 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:05:04.762747    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 13 19:05:04.762774 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:05:04.762775    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 13 19:05:04.763008 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:05:04.762758    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-gateway-1-openshift-default-dockercfg-zh96m\""
Jul 13 19:05:04.763008 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:05:04.762857    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-gw-ca-root-cert\""
Jul 13 19:05:04.768988 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:05:04.768967    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc88d7be-ddf6-48f4-b408-90290781fdb5-istio-envoy\") pod \"router-gateway-1-openshift-default-799f46c59b-fqc69\" (UID: \"bc88d7be-ddf6-48f4-b408-90290781fdb5\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-fqc69"
Jul 13 19:05:04.769089 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:05:04.769003    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc88d7be-ddf6-48f4-b408-90290781fdb5-istio-podinfo\") pod \"router-gateway-1-openshift-default-799f46c59b-fqc69\" (UID: \"bc88d7be-ddf6-48f4-b408-90290781fdb5\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-fqc69"
Jul 13 19:05:04.769089 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:05:04.769023    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc88d7be-ddf6-48f4-b408-90290781fdb5-istio-token\") pod \"router-gateway-1-openshift-default-799f46c59b-fqc69\" (UID: \"bc88d7be-ddf6-48f4-b408-90290781fdb5\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-fqc69"
Jul 13 19:05:04.769089 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:05:04.769040    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc88d7be-ddf6-48f4-b408-90290781fdb5-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-799f46c59b-fqc69\" (UID: \"bc88d7be-ddf6-48f4-b408-90290781fdb5\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-fqc69"
Jul 13 19:05:04.769203 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:05:04.769117    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc88d7be-ddf6-48f4-b408-90290781fdb5-istio-data\") pod \"router-gateway-1-openshift-default-799f46c59b-fqc69\" (UID: \"bc88d7be-ddf6-48f4-b408-90290781fdb5\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-fqc69"
Jul 13 19:05:04.769203 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:05:04.769141    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8tsbw\" (UniqueName: \"kubernetes.io/projected/bc88d7be-ddf6-48f4-b408-90290781fdb5-kube-api-access-8tsbw\") pod \"router-gateway-1-openshift-default-799f46c59b-fqc69\" (UID: \"bc88d7be-ddf6-48f4-b408-90290781fdb5\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-fqc69"
Jul 13 19:05:04.769203 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:05:04.769196    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc88d7be-ddf6-48f4-b408-90290781fdb5-credential-socket\") pod \"router-gateway-1-openshift-default-799f46c59b-fqc69\" (UID: \"bc88d7be-ddf6-48f4-b408-90290781fdb5\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-fqc69"
Jul 13 19:05:04.769298 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:05:04.769214    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc88d7be-ddf6-48f4-b408-90290781fdb5-workload-socket\") pod \"router-gateway-1-openshift-default-799f46c59b-fqc69\" (UID: \"bc88d7be-ddf6-48f4-b408-90290781fdb5\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-fqc69"
Jul 13 19:05:04.769298 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:05:04.769269    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc88d7be-ddf6-48f4-b408-90290781fdb5-workload-certs\") pod \"router-gateway-1-openshift-default-799f46c59b-fqc69\" (UID: \"bc88d7be-ddf6-48f4-b408-90290781fdb5\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-fqc69"
Jul 13 19:05:04.773735 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:05:04.773715    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-fqc69"]
Jul 13 19:05:04.870211 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:05:04.870179    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc88d7be-ddf6-48f4-b408-90290781fdb5-istio-envoy\") pod \"router-gateway-1-openshift-default-799f46c59b-fqc69\" (UID: \"bc88d7be-ddf6-48f4-b408-90290781fdb5\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-fqc69"
Jul 13 19:05:04.870363 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:05:04.870219    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc88d7be-ddf6-48f4-b408-90290781fdb5-istio-podinfo\") pod \"router-gateway-1-openshift-default-799f46c59b-fqc69\" (UID: \"bc88d7be-ddf6-48f4-b408-90290781fdb5\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-fqc69"
Jul 13 19:05:04.870363 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:05:04.870239    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc88d7be-ddf6-48f4-b408-90290781fdb5-istio-token\") pod \"router-gateway-1-openshift-default-799f46c59b-fqc69\" (UID: \"bc88d7be-ddf6-48f4-b408-90290781fdb5\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-fqc69"
Jul 13 19:05:04.870363 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:05:04.870256    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc88d7be-ddf6-48f4-b408-90290781fdb5-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-799f46c59b-fqc69\" (UID: \"bc88d7be-ddf6-48f4-b408-90290781fdb5\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-fqc69"
Jul 13 19:05:04.870363 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:05:04.870289    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc88d7be-ddf6-48f4-b408-90290781fdb5-istio-data\") pod \"router-gateway-1-openshift-default-799f46c59b-fqc69\" (UID: \"bc88d7be-ddf6-48f4-b408-90290781fdb5\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-fqc69"
Jul 13 19:05:04.870363 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:05:04.870306    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8tsbw\" (UniqueName: \"kubernetes.io/projected/bc88d7be-ddf6-48f4-b408-90290781fdb5-kube-api-access-8tsbw\") pod \"router-gateway-1-openshift-default-799f46c59b-fqc69\" (UID: \"bc88d7be-ddf6-48f4-b408-90290781fdb5\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-fqc69"
Jul 13 19:05:04.870363 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:05:04.870346    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc88d7be-ddf6-48f4-b408-90290781fdb5-credential-socket\") pod \"router-gateway-1-openshift-default-799f46c59b-fqc69\" (UID: \"bc88d7be-ddf6-48f4-b408-90290781fdb5\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-fqc69"
Jul 13 19:05:04.870753 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:05:04.870373    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc88d7be-ddf6-48f4-b408-90290781fdb5-workload-socket\") pod \"router-gateway-1-openshift-default-799f46c59b-fqc69\" (UID: \"bc88d7be-ddf6-48f4-b408-90290781fdb5\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-fqc69"
Jul 13 19:05:04.870753 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:05:04.870401    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc88d7be-ddf6-48f4-b408-90290781fdb5-workload-certs\") pod \"router-gateway-1-openshift-default-799f46c59b-fqc69\" (UID: \"bc88d7be-ddf6-48f4-b408-90290781fdb5\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-fqc69"
Jul 13 19:05:04.870858 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:05:04.870782    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc88d7be-ddf6-48f4-b408-90290781fdb5-istio-data\") pod \"router-gateway-1-openshift-default-799f46c59b-fqc69\" (UID: \"bc88d7be-ddf6-48f4-b408-90290781fdb5\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-fqc69"
Jul 13 19:05:04.870858 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:05:04.870815    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc88d7be-ddf6-48f4-b408-90290781fdb5-workload-certs\") pod \"router-gateway-1-openshift-default-799f46c59b-fqc69\" (UID: \"bc88d7be-ddf6-48f4-b408-90290781fdb5\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-fqc69"
Jul 13 19:05:04.870967 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:05:04.870870    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc88d7be-ddf6-48f4-b408-90290781fdb5-credential-socket\") pod \"router-gateway-1-openshift-default-799f46c59b-fqc69\" (UID: \"bc88d7be-ddf6-48f4-b408-90290781fdb5\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-fqc69"
Jul 13 19:05:04.871022 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:05:04.870960    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc88d7be-ddf6-48f4-b408-90290781fdb5-workload-socket\") pod \"router-gateway-1-openshift-default-799f46c59b-fqc69\" (UID: \"bc88d7be-ddf6-48f4-b408-90290781fdb5\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-fqc69"
Jul 13 19:05:04.871168 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:05:04.871149    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc88d7be-ddf6-48f4-b408-90290781fdb5-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-799f46c59b-fqc69\" (UID: \"bc88d7be-ddf6-48f4-b408-90290781fdb5\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-fqc69"
Jul 13 19:05:04.872619 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:05:04.872599    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc88d7be-ddf6-48f4-b408-90290781fdb5-istio-envoy\") pod \"router-gateway-1-openshift-default-799f46c59b-fqc69\" (UID: \"bc88d7be-ddf6-48f4-b408-90290781fdb5\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-fqc69"
Jul 13 19:05:04.872938 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:05:04.872921    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc88d7be-ddf6-48f4-b408-90290781fdb5-istio-podinfo\") pod \"router-gateway-1-openshift-default-799f46c59b-fqc69\" (UID: \"bc88d7be-ddf6-48f4-b408-90290781fdb5\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-fqc69"
Jul 13 19:05:04.881432 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:05:04.881398    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc88d7be-ddf6-48f4-b408-90290781fdb5-istio-token\") pod \"router-gateway-1-openshift-default-799f46c59b-fqc69\" (UID: \"bc88d7be-ddf6-48f4-b408-90290781fdb5\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-fqc69"
Jul 13 19:05:04.881548 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:05:04.881519    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8tsbw\" (UniqueName: \"kubernetes.io/projected/bc88d7be-ddf6-48f4-b408-90290781fdb5-kube-api-access-8tsbw\") pod \"router-gateway-1-openshift-default-799f46c59b-fqc69\" (UID: \"bc88d7be-ddf6-48f4-b408-90290781fdb5\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-fqc69"
Jul 13 19:05:05.092055 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:05:05.092019    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-fqc69"
Jul 13 19:05:05.249651 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:05:05.247841    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-fqc69"]
Jul 13 19:05:05.251827 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:05:05.251789    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 13 19:05:05.251935 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:05:05.251871    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 13 19:05:05.251935 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:05:05.251911    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 13 19:05:05.318811 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:05:05.318775    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-fqc69" event={"ID":"bc88d7be-ddf6-48f4-b408-90290781fdb5","Type":"ContainerStarted","Data":"7f5e1f25e7a87435e76966f27a56a4ed8cdeaf8db476463f3bfd1b846beccfe1"}
Jul 13 19:05:05.318908 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:05:05.318842    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-fqc69" event={"ID":"bc88d7be-ddf6-48f4-b408-90290781fdb5","Type":"ContainerStarted","Data":"67dbc78b664bec7062a5f20b6e2cb5f36f672a9de2a7a39c72311f30a49f361c"}
Jul 13 19:05:05.343881 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:05:05.343769    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-fqc69" podStartSLOduration=1.34375019 podStartE2EDuration="1.34375019s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 19:05:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 19:05:05.342565266 +0000 UTC m=+891.867698702" watchObservedRunningTime="2026-07-13 19:05:05.34375019 +0000 UTC m=+891.868883605"
Jul 13 19:05:06.092353 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:05:06.092313    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-fqc69"
Jul 13 19:05:06.097220 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:05:06.097193    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-fqc69"
Jul 13 19:05:06.323767 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:05:06.323737    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-fqc69"
Jul 13 19:05:06.324588 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:05:06.324570    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-fqc69"
Jul 13 19:05:13.915103 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:05:13.915075    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2bscc_2f0655bd-4a09-4314-8c1b-e30191eaec0e/ovn-acl-logging/0.log"
Jul 13 19:05:13.916684 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:05:13.916659    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2bscc_2f0655bd-4a09-4314-8c1b-e30191eaec0e/ovn-acl-logging/0.log"
Jul 13 19:05:13.996020 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:05:13.995989    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6ffc955574-p26g2"]
Jul 13 19:05:13.999921 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:05:13.999899    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6ffc955574-p26g2"
Jul 13 19:05:14.002582 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:05:14.002564    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 13 19:05:14.002735 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:05:14.002651    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-gv2sm\""
Jul 13 19:05:14.003816 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:05:14.003798    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"precise-prefix-cache-test-kserve-self-signed-certs\""
Jul 13 19:05:14.014644 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:05:14.014604    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6ffc955574-p26g2"]
Jul 13 19:05:14.054371 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:05:14.054348    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-746rz\" (UniqueName: \"kubernetes.io/projected/6eaf9d6b-03bc-43c8-9d86-1a573a3d587b-kube-api-access-746rz\") pod \"precise-prefix-cache-test-kserve-6ffc955574-p26g2\" (UID: \"6eaf9d6b-03bc-43c8-9d86-1a573a3d587b\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6ffc955574-p26g2"
Jul 13 19:05:14.054497 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:05:14.054378    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6eaf9d6b-03bc-43c8-9d86-1a573a3d587b-dshm\") pod \"precise-prefix-cache-test-kserve-6ffc955574-p26g2\" (UID: \"6eaf9d6b-03bc-43c8-9d86-1a573a3d587b\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6ffc955574-p26g2"
Jul 13 19:05:14.054497 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:05:14.054398    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6eaf9d6b-03bc-43c8-9d86-1a573a3d587b-model-cache\") pod \"precise-prefix-cache-test-kserve-6ffc955574-p26g2\" (UID: \"6eaf9d6b-03bc-43c8-9d86-1a573a3d587b\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6ffc955574-p26g2"
Jul 13 19:05:14.054497 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:05:14.054444    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6eaf9d6b-03bc-43c8-9d86-1a573a3d587b-tls-certs\") pod \"precise-prefix-cache-test-kserve-6ffc955574-p26g2\" (UID: \"6eaf9d6b-03bc-43c8-9d86-1a573a3d587b\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6ffc955574-p26g2"
Jul 13 19:05:14.054628 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:05:14.054505    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6eaf9d6b-03bc-43c8-9d86-1a573a3d587b-home\") pod \"precise-prefix-cache-test-kserve-6ffc955574-p26g2\" (UID: \"6eaf9d6b-03bc-43c8-9d86-1a573a3d587b\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6ffc955574-p26g2"
Jul 13 19:05:14.054628 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:05:14.054521    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6eaf9d6b-03bc-43c8-9d86-1a573a3d587b-kserve-provision-location\") pod \"precise-prefix-cache-test-kserve-6ffc955574-p26g2\" (UID: \"6eaf9d6b-03bc-43c8-9d86-1a573a3d587b\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6ffc955574-p26g2"
Jul 13 19:05:14.054628 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:05:14.054614    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6eaf9d6b-03bc-43c8-9d86-1a573a3d587b-tmp-dir\") pod \"precise-prefix-cache-test-kserve-6ffc955574-p26g2\" (UID: \"6eaf9d6b-03bc-43c8-9d86-1a573a3d587b\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6ffc955574-p26g2"
Jul 13 19:05:14.155125 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:05:14.155100    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6eaf9d6b-03bc-43c8-9d86-1a573a3d587b-tmp-dir\") pod \"precise-prefix-cache-test-kserve-6ffc955574-p26g2\" (UID: \"6eaf9d6b-03bc-43c8-9d86-1a573a3d587b\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6ffc955574-p26g2"
Jul 13 19:05:14.155282 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:05:14.155142    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-746rz\" (UniqueName: \"kubernetes.io/projected/6eaf9d6b-03bc-43c8-9d86-1a573a3d587b-kube-api-access-746rz\") pod \"precise-prefix-cache-test-kserve-6ffc955574-p26g2\" (UID: \"6eaf9d6b-03bc-43c8-9d86-1a573a3d587b\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6ffc955574-p26g2"
Jul 13 19:05:14.155282 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:05:14.155173    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6eaf9d6b-03bc-43c8-9d86-1a573a3d587b-dshm\") pod \"precise-prefix-cache-test-kserve-6ffc955574-p26g2\" (UID: \"6eaf9d6b-03bc-43c8-9d86-1a573a3d587b\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6ffc955574-p26g2"
Jul 13 19:05:14.155282 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:05:14.155199    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6eaf9d6b-03bc-43c8-9d86-1a573a3d587b-model-cache\") pod \"precise-prefix-cache-test-kserve-6ffc955574-p26g2\" (UID: \"6eaf9d6b-03bc-43c8-9d86-1a573a3d587b\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6ffc955574-p26g2"
Jul 13 19:05:14.155282 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:05:14.155232    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6eaf9d6b-03bc-43c8-9d86-1a573a3d587b-tls-certs\") pod \"precise-prefix-cache-test-kserve-6ffc955574-p26g2\" (UID: \"6eaf9d6b-03bc-43c8-9d86-1a573a3d587b\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6ffc955574-p26g2"
Jul 13 19:05:14.155551 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:05:14.155531    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6eaf9d6b-03bc-43c8-9d86-1a573a3d587b-tmp-dir\") pod \"precise-prefix-cache-test-kserve-6ffc955574-p26g2\" (UID: \"6eaf9d6b-03bc-43c8-9d86-1a573a3d587b\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6ffc955574-p26g2"
Jul 13 19:05:14.155618 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:05:14.155590    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6eaf9d6b-03bc-43c8-9d86-1a573a3d587b-model-cache\") pod \"precise-prefix-cache-test-kserve-6ffc955574-p26g2\" (UID: \"6eaf9d6b-03bc-43c8-9d86-1a573a3d587b\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6ffc955574-p26g2"
Jul 13 19:05:14.155833 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:05:14.155810    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6eaf9d6b-03bc-43c8-9d86-1a573a3d587b-home\") pod \"precise-prefix-cache-test-kserve-6ffc955574-p26g2\" (UID: \"6eaf9d6b-03bc-43c8-9d86-1a573a3d587b\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6ffc955574-p26g2"
Jul 13 19:05:14.155923 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:05:14.155852    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6eaf9d6b-03bc-43c8-9d86-1a573a3d587b-kserve-provision-location\") pod \"precise-prefix-cache-test-kserve-6ffc955574-p26g2\" (UID: \"6eaf9d6b-03bc-43c8-9d86-1a573a3d587b\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6ffc955574-p26g2"
Jul 13 19:05:14.156154 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:05:14.156116    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6eaf9d6b-03bc-43c8-9d86-1a573a3d587b-home\") pod \"precise-prefix-cache-test-kserve-6ffc955574-p26g2\" (UID: \"6eaf9d6b-03bc-43c8-9d86-1a573a3d587b\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6ffc955574-p26g2"
Jul 13 19:05:14.156341 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:05:14.156321    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6eaf9d6b-03bc-43c8-9d86-1a573a3d587b-kserve-provision-location\") pod \"precise-prefix-cache-test-kserve-6ffc955574-p26g2\" (UID: \"6eaf9d6b-03bc-43c8-9d86-1a573a3d587b\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6ffc955574-p26g2"
Jul 13 19:05:14.158060 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:05:14.158039    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6eaf9d6b-03bc-43c8-9d86-1a573a3d587b-dshm\") pod \"precise-prefix-cache-test-kserve-6ffc955574-p26g2\" (UID: \"6eaf9d6b-03bc-43c8-9d86-1a573a3d587b\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6ffc955574-p26g2"
Jul 13 19:05:14.158966 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:05:14.158943    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6eaf9d6b-03bc-43c8-9d86-1a573a3d587b-tls-certs\") pod \"precise-prefix-cache-test-kserve-6ffc955574-p26g2\" (UID: \"6eaf9d6b-03bc-43c8-9d86-1a573a3d587b\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6ffc955574-p26g2"
Jul 13 19:05:14.164027 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:05:14.163998    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-746rz\" (UniqueName: \"kubernetes.io/projected/6eaf9d6b-03bc-43c8-9d86-1a573a3d587b-kube-api-access-746rz\") pod \"precise-prefix-cache-test-kserve-6ffc955574-p26g2\" (UID: \"6eaf9d6b-03bc-43c8-9d86-1a573a3d587b\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6ffc955574-p26g2"
Jul 13 19:05:14.310807 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:05:14.310776    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6ffc955574-p26g2"
Jul 13 19:05:14.457982 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:05:14.457950    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6ffc955574-p26g2"]
Jul 13 19:05:14.460210 ip-10-0-128-215 kubenswrapper[2565]: W0713 19:05:14.460164    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6eaf9d6b_03bc_43c8_9d86_1a573a3d587b.slice/crio-037fe3e87bf3f36fd2d31bb1568b1f376e06a2b1a496b8be4636b268b316cb7f WatchSource:0}: Error finding container 037fe3e87bf3f36fd2d31bb1568b1f376e06a2b1a496b8be4636b268b316cb7f: Status 404 returned error can't find the container with id 037fe3e87bf3f36fd2d31bb1568b1f376e06a2b1a496b8be4636b268b316cb7f
Jul 13 19:05:15.377474 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:05:15.377435    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6ffc955574-p26g2" event={"ID":"6eaf9d6b-03bc-43c8-9d86-1a573a3d587b","Type":"ContainerStarted","Data":"bfc248efa90a6dab0462762101a4d38c0620537c54ddbca0309b00f286c8c4b5"}
Jul 13 19:05:15.377917 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:05:15.377492    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6ffc955574-p26g2" event={"ID":"6eaf9d6b-03bc-43c8-9d86-1a573a3d587b","Type":"ContainerStarted","Data":"037fe3e87bf3f36fd2d31bb1568b1f376e06a2b1a496b8be4636b268b316cb7f"}
Jul 13 19:05:18.400853 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:05:18.400819    2565 generic.go:356] "Generic (PLEG): container finished" podID="bc88d7be-ddf6-48f4-b408-90290781fdb5" containerID="7f5e1f25e7a87435e76966f27a56a4ed8cdeaf8db476463f3bfd1b846beccfe1" exitCode=0
Jul 13 19:05:18.401325 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:05:18.400875    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-fqc69" event={"ID":"bc88d7be-ddf6-48f4-b408-90290781fdb5","Type":"ContainerDied","Data":"7f5e1f25e7a87435e76966f27a56a4ed8cdeaf8db476463f3bfd1b846beccfe1"}
Jul 13 19:05:18.401325 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:05:18.401294    2565 scope.go:117] "RemoveContainer" containerID="7f5e1f25e7a87435e76966f27a56a4ed8cdeaf8db476463f3bfd1b846beccfe1"
Jul 13 19:05:18.402375 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:05:18.402345    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 13 19:05:18.402492 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:05:18.402432    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 13 19:05:18.402556 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:05:18.402492    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 13 19:05:18.402613 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:05:18.402550    2565 generic.go:356] "Generic (PLEG): container finished" podID="6eaf9d6b-03bc-43c8-9d86-1a573a3d587b" containerID="bfc248efa90a6dab0462762101a4d38c0620537c54ddbca0309b00f286c8c4b5" exitCode=0
Jul 13 19:05:18.402661 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:05:18.402617    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6ffc955574-p26g2" event={"ID":"6eaf9d6b-03bc-43c8-9d86-1a573a3d587b","Type":"ContainerDied","Data":"bfc248efa90a6dab0462762101a4d38c0620537c54ddbca0309b00f286c8c4b5"}
Jul 13 19:05:19.411473 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:05:19.411434    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-fqc69" event={"ID":"bc88d7be-ddf6-48f4-b408-90290781fdb5","Type":"ContainerStarted","Data":"ba1f6d32e456075beb1e675e4b0af155fd2aae6bfa8b2dd53fb8234f1479350f"}
Jul 13 19:05:20.092936 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:05:20.092899    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-fqc69"
Jul 13 19:05:20.097367 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:05:20.097338    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-fqc69"
Jul 13 19:05:20.421645 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:05:20.421598    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6ffc955574-p26g2" event={"ID":"6eaf9d6b-03bc-43c8-9d86-1a573a3d587b","Type":"ContainerStarted","Data":"fba6c0583a498c972ce649519893874c83d21e957fbe5725a4bcbfb16d798108"}
Jul 13 19:05:20.422164 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:05:20.422056    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-fqc69"
Jul 13 19:05:20.423234 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:05:20.423206    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-fqc69"
Jul 13 19:05:20.444762 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:05:20.444690    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6ffc955574-p26g2" podStartSLOduration=2.338672313 podStartE2EDuration="7.444670478s" totalImagesPullingTime="1.480471582s" totalInitContainerRuntime="3.625526583s" isStatefulPod=false podCreationTimestamp="2026-07-13 19:05:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 19:05:20.443825623 +0000 UTC m=+906.968959050" watchObservedRunningTime="2026-07-13 19:05:20.444670478 +0000 UTC m=+906.969803893"
Jul 13 19:05:20.506924 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:05:20.506892    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5b4jkrn"]
Jul 13 19:05:20.511656 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:05:20.511631    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5b4jkrn"
Jul 13 19:05:20.522476 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:05:20.520168    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"gw-sec2774c263d49959f50d9eebc552e13bf9-kserve-self-signed-certs\""
Jul 13 19:05:20.539848 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:05:20.539827    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5b4jkrn"]
Jul 13 19:05:20.633749 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:05:20.630175    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc0c3d21-31f7-4b3f-aec7-576a5b9c5917-dshm\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5b4jkrn\" (UID: \"fc0c3d21-31f7-4b3f-aec7-576a5b9c5917\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5b4jkrn"
Jul 13 19:05:20.633749 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:05:20.630233    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ksv4q\" (UniqueName: \"kubernetes.io/projected/fc0c3d21-31f7-4b3f-aec7-576a5b9c5917-kube-api-access-ksv4q\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5b4jkrn\" (UID: \"fc0c3d21-31f7-4b3f-aec7-576a5b9c5917\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5b4jkrn"
Jul 13 19:05:20.633749 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:05:20.630278    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc0c3d21-31f7-4b3f-aec7-576a5b9c5917-model-cache\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5b4jkrn\" (UID: \"fc0c3d21-31f7-4b3f-aec7-576a5b9c5917\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5b4jkrn"
Jul 13 19:05:20.633749 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:05:20.630334    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc0c3d21-31f7-4b3f-aec7-576a5b9c5917-tmp-dir\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5b4jkrn\" (UID: \"fc0c3d21-31f7-4b3f-aec7-576a5b9c5917\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5b4jkrn"
Jul 13 19:05:20.633749 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:05:20.630402    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc0c3d21-31f7-4b3f-aec7-576a5b9c5917-tls-certs\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5b4jkrn\" (UID: \"fc0c3d21-31f7-4b3f-aec7-576a5b9c5917\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5b4jkrn"
Jul 13 19:05:20.633749 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:05:20.630458    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc0c3d21-31f7-4b3f-aec7-576a5b9c5917-home\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5b4jkrn\" (UID: \"fc0c3d21-31f7-4b3f-aec7-576a5b9c5917\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5b4jkrn"
Jul 13 19:05:20.732123 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:05:20.732025    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc0c3d21-31f7-4b3f-aec7-576a5b9c5917-tmp-dir\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5b4jkrn\" (UID: \"fc0c3d21-31f7-4b3f-aec7-576a5b9c5917\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5b4jkrn"
Jul 13 19:05:20.732123 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:05:20.732118    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc0c3d21-31f7-4b3f-aec7-576a5b9c5917-tls-certs\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5b4jkrn\" (UID: \"fc0c3d21-31f7-4b3f-aec7-576a5b9c5917\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5b4jkrn"
Jul 13 19:05:20.732353 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:05:20.732176    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc0c3d21-31f7-4b3f-aec7-576a5b9c5917-home\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5b4jkrn\" (UID: \"fc0c3d21-31f7-4b3f-aec7-576a5b9c5917\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5b4jkrn"
Jul 13 19:05:20.732353 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:05:20.732223    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc0c3d21-31f7-4b3f-aec7-576a5b9c5917-dshm\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5b4jkrn\" (UID: \"fc0c3d21-31f7-4b3f-aec7-576a5b9c5917\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5b4jkrn"
Jul 13 19:05:20.732353 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:05:20.732259    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ksv4q\" (UniqueName: \"kubernetes.io/projected/fc0c3d21-31f7-4b3f-aec7-576a5b9c5917-kube-api-access-ksv4q\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5b4jkrn\" (UID: \"fc0c3d21-31f7-4b3f-aec7-576a5b9c5917\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5b4jkrn"
Jul 13 19:05:20.732353 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:05:20.732304    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc0c3d21-31f7-4b3f-aec7-576a5b9c5917-model-cache\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5b4jkrn\" (UID: \"fc0c3d21-31f7-4b3f-aec7-576a5b9c5917\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5b4jkrn"
Jul 13 19:05:20.732774 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:05:20.732748    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc0c3d21-31f7-4b3f-aec7-576a5b9c5917-model-cache\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5b4jkrn\" (UID: \"fc0c3d21-31f7-4b3f-aec7-576a5b9c5917\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5b4jkrn"
Jul 13 19:05:20.733045 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:05:20.733021    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc0c3d21-31f7-4b3f-aec7-576a5b9c5917-tmp-dir\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5b4jkrn\" (UID: \"fc0c3d21-31f7-4b3f-aec7-576a5b9c5917\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5b4jkrn"
Jul 13 19:05:20.733769 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:05:20.733740    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc0c3d21-31f7-4b3f-aec7-576a5b9c5917-home\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5b4jkrn\" (UID: \"fc0c3d21-31f7-4b3f-aec7-576a5b9c5917\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5b4jkrn"
Jul 13 19:05:20.738740 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:05:20.736257    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc0c3d21-31f7-4b3f-aec7-576a5b9c5917-dshm\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5b4jkrn\" (UID: \"fc0c3d21-31f7-4b3f-aec7-576a5b9c5917\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5b4jkrn"
Jul 13 19:05:20.738740 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:05:20.736305    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc0c3d21-31f7-4b3f-aec7-576a5b9c5917-tls-certs\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5b4jkrn\" (UID: \"fc0c3d21-31f7-4b3f-aec7-576a5b9c5917\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5b4jkrn"
Jul 13 19:05:20.748924 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:05:20.748899    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ksv4q\" (UniqueName: \"kubernetes.io/projected/fc0c3d21-31f7-4b3f-aec7-576a5b9c5917-kube-api-access-ksv4q\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5b4jkrn\" (UID: \"fc0c3d21-31f7-4b3f-aec7-576a5b9c5917\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5b4jkrn"
Jul 13 19:05:20.841173 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:05:20.841139    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5b4jkrn"
Jul 13 19:05:20.999280 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:05:20.999257    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5b4jkrn"]
Jul 13 19:05:21.000985 ip-10-0-128-215 kubenswrapper[2565]: W0713 19:05:21.000950    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc0c3d21_31f7_4b3f_aec7_576a5b9c5917.slice/crio-49a080b5b6c7c07e244dd3e26f47881f5071b1edf9401c6935f3db9100709b64 WatchSource:0}: Error finding container 49a080b5b6c7c07e244dd3e26f47881f5071b1edf9401c6935f3db9100709b64: Status 404 returned error can't find the container with id 49a080b5b6c7c07e244dd3e26f47881f5071b1edf9401c6935f3db9100709b64
Jul 13 19:05:21.311648 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:05:21.311623    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6ffc955574-p26g2"
Jul 13 19:05:21.434329 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:05:21.434289    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5b4jkrn" event={"ID":"fc0c3d21-31f7-4b3f-aec7-576a5b9c5917","Type":"ContainerStarted","Data":"3ff1d920bacb23b3020b769b57cb9c84219140c7e8fd59ccc1e7dd16b9648f6f"}
Jul 13 19:05:21.434884 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:05:21.434360    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5b4jkrn" event={"ID":"fc0c3d21-31f7-4b3f-aec7-576a5b9c5917","Type":"ContainerStarted","Data":"49a080b5b6c7c07e244dd3e26f47881f5071b1edf9401c6935f3db9100709b64"}
Jul 13 19:05:21.457458 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:05:21.457409    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5b4jkrn" podStartSLOduration=1.457393653 podStartE2EDuration="1.457393653s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 19:05:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 19:05:21.454753685 +0000 UTC m=+907.979887114" watchObservedRunningTime="2026-07-13 19:05:21.457393653 +0000 UTC m=+907.982527069"
Jul 13 19:05:21.841318 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:05:21.841266    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5b4jkrn"
Jul 13 19:05:24.311665 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:05:24.311629    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6ffc955574-p26g2"
Jul 13 19:05:24.335256 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:05:24.335222    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6ffc955574-p26g2"
Jul 13 19:05:24.470393 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:05:24.470363    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6ffc955574-p26g2"
Jul 13 19:05:26.546310 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:05:26.195581    2565 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-655f7c58b4-8g97n container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": read tcp 10.134.0.2:47062->10.134.0.33:15021: read: connection reset by peer" start-of-body=
Jul 13 19:05:26.546310 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:05:26.195640    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-8g97n" podUID="87a48a7a-6d63-43f0-8bab-7e29c1a6979a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": read tcp 10.134.0.2:47062->10.134.0.33:15021: read: connection reset by peer"
Jul 13 19:05:26.546310 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:05:26.465690    2565 generic.go:356] "Generic (PLEG): container finished" podID="87a48a7a-6d63-43f0-8bab-7e29c1a6979a" containerID="dd3948ff3e8e5c8ecb3d67eb4faf9886831fcaafebef990c0de293d4aa95f089" exitCode=0
Jul 13 19:05:26.546310 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:05:26.465800    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-8g97n" event={"ID":"87a48a7a-6d63-43f0-8bab-7e29c1a6979a","Type":"ContainerDied","Data":"dd3948ff3e8e5c8ecb3d67eb4faf9886831fcaafebef990c0de293d4aa95f089"}
Jul 13 19:05:26.546310 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:05:26.466214    2565 scope.go:117] "RemoveContainer" containerID="dd3948ff3e8e5c8ecb3d67eb4faf9886831fcaafebef990c0de293d4aa95f089"
Jul 13 19:05:26.546310 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:05:26.467143    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 13 19:05:26.546310 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:05:26.467200    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 13 19:05:26.546310 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:05:26.467232    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 13 19:05:27.472639 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:05:27.472605    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-8g97n" event={"ID":"87a48a7a-6d63-43f0-8bab-7e29c1a6979a","Type":"ContainerStarted","Data":"4eb92dc57c51186beb2db648074a6552379a677ccbaf1c815310aa15999081b0"}
Jul 13 19:05:27.523801 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:05:27.523761    2565 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-655f7c58b4-8g97n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body=
Jul 13 19:05:27.524007 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:05:27.523865    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-8g97n" podUID="87a48a7a-6d63-43f0-8bab-7e29c1a6979a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused"
Jul 13 19:05:27.524170 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:05:27.523315    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-8g97n"
Jul 13 19:05:28.244635 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:05:28.244597    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-fqc69"]
Jul 13 19:05:28.246124 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:05:28.246060    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-fqc69" podUID="bc88d7be-ddf6-48f4-b408-90290781fdb5" containerName="istio-proxy" containerID="cri-o://ba1f6d32e456075beb1e675e4b0af155fd2aae6bfa8b2dd53fb8234f1479350f" gracePeriod=30
Jul 13 19:05:28.524184 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:05:28.524145    2565 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-655f7c58b4-8g97n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body=
Jul 13 19:05:28.524439 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:05:28.524412    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-8g97n" podUID="87a48a7a-6d63-43f0-8bab-7e29c1a6979a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused"
Jul 13 19:05:29.524467 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:05:29.524017    2565 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-655f7c58b4-8g97n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body=
Jul 13 19:05:29.524467 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:05:29.524080    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-8g97n" podUID="87a48a7a-6d63-43f0-8bab-7e29c1a6979a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused"
Jul 13 19:05:30.527114 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:05:30.527077    2565 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-655f7c58b4-8g97n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body=
Jul 13 19:05:30.527736 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:05:30.527677    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-8g97n" podUID="87a48a7a-6d63-43f0-8bab-7e29c1a6979a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused"
Jul 13 19:05:30.842158 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:05:30.842059    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5b4jkrn"
Jul 13 19:05:30.854738 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:05:30.854692    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5b4jkrn"
Jul 13 19:05:31.494038 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:05:31.493951    2565 generic.go:356] "Generic (PLEG): container finished" podID="bc88d7be-ddf6-48f4-b408-90290781fdb5" containerID="ba1f6d32e456075beb1e675e4b0af155fd2aae6bfa8b2dd53fb8234f1479350f" exitCode=0
Jul 13 19:05:31.494038 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:05:31.494018    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-fqc69" event={"ID":"bc88d7be-ddf6-48f4-b408-90290781fdb5","Type":"ContainerDied","Data":"ba1f6d32e456075beb1e675e4b0af155fd2aae6bfa8b2dd53fb8234f1479350f"}
Jul 13 19:05:31.494226 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:05:31.494068    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-fqc69" event={"ID":"bc88d7be-ddf6-48f4-b408-90290781fdb5","Type":"ContainerDied","Data":"67dbc78b664bec7062a5f20b6e2cb5f36f672a9de2a7a39c72311f30a49f361c"}
Jul 13 19:05:31.494226 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:05:31.494085    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="67dbc78b664bec7062a5f20b6e2cb5f36f672a9de2a7a39c72311f30a49f361c"
Jul 13 19:05:31.494226 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:05:31.494099    2565 scope.go:117] "RemoveContainer" containerID="7f5e1f25e7a87435e76966f27a56a4ed8cdeaf8db476463f3bfd1b846beccfe1"
Jul 13 19:05:31.506076 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:05:31.506057    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5b4jkrn"
Jul 13 19:05:31.509860 ip-10-0-128-215 kubenswrapper[2565]: E0713 19:05:31.509836    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f5e1f25e7a87435e76966f27a56a4ed8cdeaf8db476463f3bfd1b846beccfe1\": container with ID starting with 7f5e1f25e7a87435e76966f27a56a4ed8cdeaf8db476463f3bfd1b846beccfe1 not found: ID does not exist" containerID="7f5e1f25e7a87435e76966f27a56a4ed8cdeaf8db476463f3bfd1b846beccfe1"
Jul 13 19:05:31.511684 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:05:31.511667    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-fqc69"
Jul 13 19:05:31.538892 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:05:31.538832    2565 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-655f7c58b4-8g97n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body=
Jul 13 19:05:31.539348 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:05:31.538891    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-8g97n" podUID="87a48a7a-6d63-43f0-8bab-7e29c1a6979a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused"
Jul 13 19:05:31.663953 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:05:31.663864    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc88d7be-ddf6-48f4-b408-90290781fdb5-istio-podinfo\") pod \"bc88d7be-ddf6-48f4-b408-90290781fdb5\" (UID: \"bc88d7be-ddf6-48f4-b408-90290781fdb5\") "
Jul 13 19:05:31.663953 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:05:31.663937    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc88d7be-ddf6-48f4-b408-90290781fdb5-istiod-ca-cert\") pod \"bc88d7be-ddf6-48f4-b408-90290781fdb5\" (UID: \"bc88d7be-ddf6-48f4-b408-90290781fdb5\") "
Jul 13 19:05:31.664183 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:05:31.663958    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc88d7be-ddf6-48f4-b408-90290781fdb5-workload-socket\") pod \"bc88d7be-ddf6-48f4-b408-90290781fdb5\" (UID: \"bc88d7be-ddf6-48f4-b408-90290781fdb5\") "
Jul 13 19:05:31.664183 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:05:31.663985    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc88d7be-ddf6-48f4-b408-90290781fdb5-istio-envoy\") pod \"bc88d7be-ddf6-48f4-b408-90290781fdb5\" (UID: \"bc88d7be-ddf6-48f4-b408-90290781fdb5\") "
Jul 13 19:05:31.664183 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:05:31.664004    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc88d7be-ddf6-48f4-b408-90290781fdb5-workload-certs\") pod \"bc88d7be-ddf6-48f4-b408-90290781fdb5\" (UID: \"bc88d7be-ddf6-48f4-b408-90290781fdb5\") "
Jul 13 19:05:31.664183 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:05:31.664061    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc88d7be-ddf6-48f4-b408-90290781fdb5-credential-socket\") pod \"bc88d7be-ddf6-48f4-b408-90290781fdb5\" (UID: \"bc88d7be-ddf6-48f4-b408-90290781fdb5\") "
Jul 13 19:05:31.664183 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:05:31.664082    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc88d7be-ddf6-48f4-b408-90290781fdb5-istio-data\") pod \"bc88d7be-ddf6-48f4-b408-90290781fdb5\" (UID: \"bc88d7be-ddf6-48f4-b408-90290781fdb5\") "
Jul 13 19:05:31.664183 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:05:31.664098    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8tsbw\" (UniqueName: \"kubernetes.io/projected/bc88d7be-ddf6-48f4-b408-90290781fdb5-kube-api-access-8tsbw\") pod \"bc88d7be-ddf6-48f4-b408-90290781fdb5\" (UID: \"bc88d7be-ddf6-48f4-b408-90290781fdb5\") "
Jul 13 19:05:31.664455 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:05:31.664266    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc88d7be-ddf6-48f4-b408-90290781fdb5-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "bc88d7be-ddf6-48f4-b408-90290781fdb5" (UID: "bc88d7be-ddf6-48f4-b408-90290781fdb5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:05:31.664455 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:05:31.664357    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc88d7be-ddf6-48f4-b408-90290781fdb5-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "bc88d7be-ddf6-48f4-b408-90290781fdb5" (UID: "bc88d7be-ddf6-48f4-b408-90290781fdb5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:05:31.664455 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:05:31.664366    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc88d7be-ddf6-48f4-b408-90290781fdb5-istio-data" (OuterVolumeSpecName: "istio-data") pod "bc88d7be-ddf6-48f4-b408-90290781fdb5" (UID: "bc88d7be-ddf6-48f4-b408-90290781fdb5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:05:31.664455 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:05:31.664369    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc88d7be-ddf6-48f4-b408-90290781fdb5-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "bc88d7be-ddf6-48f4-b408-90290781fdb5" (UID: "bc88d7be-ddf6-48f4-b408-90290781fdb5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 19:05:31.664455 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:05:31.664411    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc88d7be-ddf6-48f4-b408-90290781fdb5-istio-token\") pod \"bc88d7be-ddf6-48f4-b408-90290781fdb5\" (UID: \"bc88d7be-ddf6-48f4-b408-90290781fdb5\") "
Jul 13 19:05:31.664455 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:05:31.664439    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc88d7be-ddf6-48f4-b408-90290781fdb5-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "bc88d7be-ddf6-48f4-b408-90290781fdb5" (UID: "bc88d7be-ddf6-48f4-b408-90290781fdb5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:05:31.664870 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:05:31.664850    2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc88d7be-ddf6-48f4-b408-90290781fdb5-istiod-ca-cert\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 13 19:05:31.664992 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:05:31.664875    2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc88d7be-ddf6-48f4-b408-90290781fdb5-workload-socket\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 13 19:05:31.664992 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:05:31.664888    2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc88d7be-ddf6-48f4-b408-90290781fdb5-workload-certs\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 13 19:05:31.664992 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:05:31.664900    2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc88d7be-ddf6-48f4-b408-90290781fdb5-credential-socket\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 13 19:05:31.664992 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:05:31.664913    2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc88d7be-ddf6-48f4-b408-90290781fdb5-istio-data\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 13 19:05:31.666516 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:05:31.666477    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bc88d7be-ddf6-48f4-b408-90290781fdb5-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "bc88d7be-ddf6-48f4-b408-90290781fdb5" (UID: "bc88d7be-ddf6-48f4-b408-90290781fdb5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 13 19:05:31.666516 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:05:31.666503    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc88d7be-ddf6-48f4-b408-90290781fdb5-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "bc88d7be-ddf6-48f4-b408-90290781fdb5" (UID: "bc88d7be-ddf6-48f4-b408-90290781fdb5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:05:31.666751 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:05:31.666599    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc88d7be-ddf6-48f4-b408-90290781fdb5-istio-token" (OuterVolumeSpecName: "istio-token") pod "bc88d7be-ddf6-48f4-b408-90290781fdb5" (UID: "bc88d7be-ddf6-48f4-b408-90290781fdb5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 19:05:31.666813 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:05:31.666794    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc88d7be-ddf6-48f4-b408-90290781fdb5-kube-api-access-8tsbw" (OuterVolumeSpecName: "kube-api-access-8tsbw") pod "bc88d7be-ddf6-48f4-b408-90290781fdb5" (UID: "bc88d7be-ddf6-48f4-b408-90290781fdb5"). InnerVolumeSpecName "kube-api-access-8tsbw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 19:05:31.765428 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:05:31.765391    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8tsbw\" (UniqueName: \"kubernetes.io/projected/bc88d7be-ddf6-48f4-b408-90290781fdb5-kube-api-access-8tsbw\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 13 19:05:31.765428 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:05:31.765424    2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc88d7be-ddf6-48f4-b408-90290781fdb5-istio-token\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 13 19:05:31.765428 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:05:31.765433    2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc88d7be-ddf6-48f4-b408-90290781fdb5-istio-podinfo\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 13 19:05:31.765660 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:05:31.765441    2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc88d7be-ddf6-48f4-b408-90290781fdb5-istio-envoy\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 13 19:05:32.501294 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:05:32.501209    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-fqc69"
Jul 13 19:05:32.521306 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:05:32.521272    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-fqc69"]
Jul 13 19:05:32.527660 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:05:32.527631    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-fqc69"]
Jul 13 19:05:32.545962 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:05:32.545942    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-8g97n"
Jul 13 19:05:32.546414 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:05:32.546398    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-8g97n"
Jul 13 19:05:32.547068 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:05:32.547045    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-8g97n"
Jul 13 19:05:34.049101 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:05:34.049069    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bc88d7be-ddf6-48f4-b408-90290781fdb5" path="/var/lib/kubelet/pods/bc88d7be-ddf6-48f4-b408-90290781fdb5/volumes"
Jul 13 19:05:50.478012 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:05:50.477977    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5b4jkrn"]
Jul 13 19:06:05.498631 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:06:05.498566    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5b4jkrn" podUID="fc0c3d21-31f7-4b3f-aec7-576a5b9c5917" containerName="main" containerID="cri-o://3ff1d920bacb23b3020b769b57cb9c84219140c7e8fd59ccc1e7dd16b9648f6f" gracePeriod=45
Jul 13 19:06:05.502916 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:06:05.502887    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5b4jkrn" podUID="fc0c3d21-31f7-4b3f-aec7-576a5b9c5917" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/health\": EOF"
Jul 13 19:06:05.675649 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:06:05.675622    2565 generic.go:356] "Generic (PLEG): container finished" podID="fc0c3d21-31f7-4b3f-aec7-576a5b9c5917" containerID="3ff1d920bacb23b3020b769b57cb9c84219140c7e8fd59ccc1e7dd16b9648f6f" exitCode=0
Jul 13 19:06:05.675797 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:06:05.675692    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5b4jkrn" event={"ID":"fc0c3d21-31f7-4b3f-aec7-576a5b9c5917","Type":"ContainerDied","Data":"3ff1d920bacb23b3020b769b57cb9c84219140c7e8fd59ccc1e7dd16b9648f6f"}
Jul 13 19:06:05.758959 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:06:05.758897    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5b4jkrn"
Jul 13 19:06:05.860316 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:06:05.860274    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc0c3d21-31f7-4b3f-aec7-576a5b9c5917-dshm\") pod \"fc0c3d21-31f7-4b3f-aec7-576a5b9c5917\" (UID: \"fc0c3d21-31f7-4b3f-aec7-576a5b9c5917\") "
Jul 13 19:06:05.860485 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:06:05.860328    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc0c3d21-31f7-4b3f-aec7-576a5b9c5917-home\") pod \"fc0c3d21-31f7-4b3f-aec7-576a5b9c5917\" (UID: \"fc0c3d21-31f7-4b3f-aec7-576a5b9c5917\") "
Jul 13 19:06:05.860485 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:06:05.860349    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc0c3d21-31f7-4b3f-aec7-576a5b9c5917-model-cache\") pod \"fc0c3d21-31f7-4b3f-aec7-576a5b9c5917\" (UID: \"fc0c3d21-31f7-4b3f-aec7-576a5b9c5917\") "
Jul 13 19:06:05.860485 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:06:05.860393    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ksv4q\" (UniqueName: \"kubernetes.io/projected/fc0c3d21-31f7-4b3f-aec7-576a5b9c5917-kube-api-access-ksv4q\") pod \"fc0c3d21-31f7-4b3f-aec7-576a5b9c5917\" (UID: \"fc0c3d21-31f7-4b3f-aec7-576a5b9c5917\") "
Jul 13 19:06:05.860485 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:06:05.860415    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc0c3d21-31f7-4b3f-aec7-576a5b9c5917-tmp-dir\") pod \"fc0c3d21-31f7-4b3f-aec7-576a5b9c5917\" (UID: \"fc0c3d21-31f7-4b3f-aec7-576a5b9c5917\") "
Jul 13 19:06:05.860724 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:06:05.860554    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc0c3d21-31f7-4b3f-aec7-576a5b9c5917-tls-certs\") pod \"fc0c3d21-31f7-4b3f-aec7-576a5b9c5917\" (UID: \"fc0c3d21-31f7-4b3f-aec7-576a5b9c5917\") "
Jul 13 19:06:05.860724 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:06:05.860601    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc0c3d21-31f7-4b3f-aec7-576a5b9c5917-home" (OuterVolumeSpecName: "home") pod "fc0c3d21-31f7-4b3f-aec7-576a5b9c5917" (UID: "fc0c3d21-31f7-4b3f-aec7-576a5b9c5917"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:06:05.860724 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:06:05.860635    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc0c3d21-31f7-4b3f-aec7-576a5b9c5917-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "fc0c3d21-31f7-4b3f-aec7-576a5b9c5917" (UID: "fc0c3d21-31f7-4b3f-aec7-576a5b9c5917"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:06:05.860724 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:06:05.860649    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc0c3d21-31f7-4b3f-aec7-576a5b9c5917-model-cache" (OuterVolumeSpecName: "model-cache") pod "fc0c3d21-31f7-4b3f-aec7-576a5b9c5917" (UID: "fc0c3d21-31f7-4b3f-aec7-576a5b9c5917"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:06:05.860938 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:06:05.860850    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc0c3d21-31f7-4b3f-aec7-576a5b9c5917-tmp-dir\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 13 19:06:05.860938 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:06:05.860862    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc0c3d21-31f7-4b3f-aec7-576a5b9c5917-home\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 13 19:06:05.860938 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:06:05.860874    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc0c3d21-31f7-4b3f-aec7-576a5b9c5917-model-cache\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 13 19:06:05.862669 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:06:05.862640    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc0c3d21-31f7-4b3f-aec7-576a5b9c5917-dshm" (OuterVolumeSpecName: "dshm") pod "fc0c3d21-31f7-4b3f-aec7-576a5b9c5917" (UID: "fc0c3d21-31f7-4b3f-aec7-576a5b9c5917"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:06:05.862795 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:06:05.862716    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc0c3d21-31f7-4b3f-aec7-576a5b9c5917-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fc0c3d21-31f7-4b3f-aec7-576a5b9c5917" (UID: "fc0c3d21-31f7-4b3f-aec7-576a5b9c5917"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 19:06:05.862795 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:06:05.862753    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc0c3d21-31f7-4b3f-aec7-576a5b9c5917-kube-api-access-ksv4q" (OuterVolumeSpecName: "kube-api-access-ksv4q") pod "fc0c3d21-31f7-4b3f-aec7-576a5b9c5917" (UID: "fc0c3d21-31f7-4b3f-aec7-576a5b9c5917"). InnerVolumeSpecName "kube-api-access-ksv4q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 19:06:05.961545 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:06:05.961519    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ksv4q\" (UniqueName: \"kubernetes.io/projected/fc0c3d21-31f7-4b3f-aec7-576a5b9c5917-kube-api-access-ksv4q\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 13 19:06:05.961545 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:06:05.961543    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc0c3d21-31f7-4b3f-aec7-576a5b9c5917-tls-certs\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 13 19:06:05.961682 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:06:05.961553    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc0c3d21-31f7-4b3f-aec7-576a5b9c5917-dshm\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 13 19:06:06.681757 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:06:06.681729    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5b4jkrn"
Jul 13 19:06:06.681757 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:06:06.681752    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5b4jkrn" event={"ID":"fc0c3d21-31f7-4b3f-aec7-576a5b9c5917","Type":"ContainerDied","Data":"49a080b5b6c7c07e244dd3e26f47881f5071b1edf9401c6935f3db9100709b64"}
Jul 13 19:06:06.682586 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:06:06.681847    2565 scope.go:117] "RemoveContainer" containerID="3ff1d920bacb23b3020b769b57cb9c84219140c7e8fd59ccc1e7dd16b9648f6f"
Jul 13 19:06:06.698303 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:06:06.698279    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5b4jkrn"]
Jul 13 19:06:06.700097 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:06:06.700076    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7bc8dd6c5b4jkrn"]
Jul 13 19:06:07.690792 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:06:07.690762    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/auth-enabled-test-kserve-7dfddf4646-c84nz"]
Jul 13 19:06:07.691259 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:06:07.691241    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc88d7be-ddf6-48f4-b408-90290781fdb5" containerName="istio-proxy"
Jul 13 19:06:07.691259 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:06:07.691254    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc88d7be-ddf6-48f4-b408-90290781fdb5" containerName="istio-proxy"
Jul 13 19:06:07.691380 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:06:07.691271    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc0c3d21-31f7-4b3f-aec7-576a5b9c5917" containerName="main"
Jul 13 19:06:07.691380 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:06:07.691276    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc0c3d21-31f7-4b3f-aec7-576a5b9c5917" containerName="main"
Jul 13 19:06:07.691380 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:06:07.691288    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc88d7be-ddf6-48f4-b408-90290781fdb5" containerName="istio-proxy"
Jul 13 19:06:07.691380 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:06:07.691294    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc88d7be-ddf6-48f4-b408-90290781fdb5" containerName="istio-proxy"
Jul 13 19:06:07.691380 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:06:07.691353    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="bc88d7be-ddf6-48f4-b408-90290781fdb5" containerName="istio-proxy"
Jul 13 19:06:07.691380 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:06:07.691361    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc0c3d21-31f7-4b3f-aec7-576a5b9c5917" containerName="main"
Jul 13 19:06:07.691686 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:06:07.691505    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="bc88d7be-ddf6-48f4-b408-90290781fdb5" containerName="istio-proxy"
Jul 13 19:06:07.694388 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:06:07.694368    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-7dfddf4646-c84nz"
Jul 13 19:06:07.697079 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:06:07.697058    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"auth-enabled-test-kserve-self-signed-certs\""
Jul 13 19:06:07.703034 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:06:07.703013    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/auth-enabled-test-kserve-7dfddf4646-c84nz"]
Jul 13 19:06:07.776350 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:06:07.776320    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ee3e4b01-2183-44a8-8cc5-140f52eca09f-kserve-provision-location\") pod \"auth-enabled-test-kserve-7dfddf4646-c84nz\" (UID: \"ee3e4b01-2183-44a8-8cc5-140f52eca09f\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-7dfddf4646-c84nz"
Jul 13 19:06:07.776504 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:06:07.776360    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee3e4b01-2183-44a8-8cc5-140f52eca09f-dshm\") pod \"auth-enabled-test-kserve-7dfddf4646-c84nz\" (UID: \"ee3e4b01-2183-44a8-8cc5-140f52eca09f\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-7dfddf4646-c84nz"
Jul 13 19:06:07.776504 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:06:07.776409    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4lnl\" (UniqueName: \"kubernetes.io/projected/ee3e4b01-2183-44a8-8cc5-140f52eca09f-kube-api-access-w4lnl\") pod \"auth-enabled-test-kserve-7dfddf4646-c84nz\" (UID: \"ee3e4b01-2183-44a8-8cc5-140f52eca09f\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-7dfddf4646-c84nz"
Jul 13 19:06:07.776504 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:06:07.776452    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee3e4b01-2183-44a8-8cc5-140f52eca09f-home\") pod \"auth-enabled-test-kserve-7dfddf4646-c84nz\" (UID: \"ee3e4b01-2183-44a8-8cc5-140f52eca09f\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-7dfddf4646-c84nz"
Jul 13 19:06:07.776504 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:06:07.776471    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee3e4b01-2183-44a8-8cc5-140f52eca09f-model-cache\") pod \"auth-enabled-test-kserve-7dfddf4646-c84nz\" (UID: \"ee3e4b01-2183-44a8-8cc5-140f52eca09f\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-7dfddf4646-c84nz"
Jul 13 19:06:07.776504 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:06:07.776492    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee3e4b01-2183-44a8-8cc5-140f52eca09f-tls-certs\") pod \"auth-enabled-test-kserve-7dfddf4646-c84nz\" (UID: \"ee3e4b01-2183-44a8-8cc5-140f52eca09f\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-7dfddf4646-c84nz"
Jul 13 19:06:07.776725 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:06:07.776525    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee3e4b01-2183-44a8-8cc5-140f52eca09f-tmp-dir\") pod \"auth-enabled-test-kserve-7dfddf4646-c84nz\" (UID: \"ee3e4b01-2183-44a8-8cc5-140f52eca09f\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-7dfddf4646-c84nz"
Jul 13 19:06:07.878025 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:06:07.877997    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ee3e4b01-2183-44a8-8cc5-140f52eca09f-kserve-provision-location\") pod \"auth-enabled-test-kserve-7dfddf4646-c84nz\" (UID: \"ee3e4b01-2183-44a8-8cc5-140f52eca09f\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-7dfddf4646-c84nz"
Jul 13 19:06:07.878210 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:06:07.878034    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee3e4b01-2183-44a8-8cc5-140f52eca09f-dshm\") pod \"auth-enabled-test-kserve-7dfddf4646-c84nz\" (UID: \"ee3e4b01-2183-44a8-8cc5-140f52eca09f\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-7dfddf4646-c84nz"
Jul 13 19:06:07.878210 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:06:07.878076    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4lnl\" (UniqueName: \"kubernetes.io/projected/ee3e4b01-2183-44a8-8cc5-140f52eca09f-kube-api-access-w4lnl\") pod \"auth-enabled-test-kserve-7dfddf4646-c84nz\" (UID: \"ee3e4b01-2183-44a8-8cc5-140f52eca09f\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-7dfddf4646-c84nz"
Jul 13 19:06:07.878210 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:06:07.878120    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee3e4b01-2183-44a8-8cc5-140f52eca09f-home\") pod \"auth-enabled-test-kserve-7dfddf4646-c84nz\" (UID: \"ee3e4b01-2183-44a8-8cc5-140f52eca09f\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-7dfddf4646-c84nz"
Jul 13 19:06:07.878210 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:06:07.878150    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee3e4b01-2183-44a8-8cc5-140f52eca09f-model-cache\") pod \"auth-enabled-test-kserve-7dfddf4646-c84nz\" (UID: \"ee3e4b01-2183-44a8-8cc5-140f52eca09f\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-7dfddf4646-c84nz"
Jul 13 19:06:07.878210 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:06:07.878179    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee3e4b01-2183-44a8-8cc5-140f52eca09f-tls-certs\") pod \"auth-enabled-test-kserve-7dfddf4646-c84nz\" (UID: \"ee3e4b01-2183-44a8-8cc5-140f52eca09f\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-7dfddf4646-c84nz"
Jul 13 19:06:07.878476 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:06:07.878218    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee3e4b01-2183-44a8-8cc5-140f52eca09f-tmp-dir\") pod \"auth-enabled-test-kserve-7dfddf4646-c84nz\" (UID: \"ee3e4b01-2183-44a8-8cc5-140f52eca09f\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-7dfddf4646-c84nz"
Jul 13 19:06:07.878476 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:06:07.878448    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ee3e4b01-2183-44a8-8cc5-140f52eca09f-kserve-provision-location\") pod \"auth-enabled-test-kserve-7dfddf4646-c84nz\" (UID: \"ee3e4b01-2183-44a8-8cc5-140f52eca09f\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-7dfddf4646-c84nz"
Jul 13 19:06:07.878596 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:06:07.878548    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee3e4b01-2183-44a8-8cc5-140f52eca09f-home\") pod \"auth-enabled-test-kserve-7dfddf4646-c84nz\" (UID: \"ee3e4b01-2183-44a8-8cc5-140f52eca09f\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-7dfddf4646-c84nz"
Jul 13 19:06:07.878596 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:06:07.878584    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee3e4b01-2183-44a8-8cc5-140f52eca09f-model-cache\") pod \"auth-enabled-test-kserve-7dfddf4646-c84nz\" (UID: \"ee3e4b01-2183-44a8-8cc5-140f52eca09f\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-7dfddf4646-c84nz"
Jul 13 19:06:07.878726 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:06:07.878639    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee3e4b01-2183-44a8-8cc5-140f52eca09f-tmp-dir\") pod \"auth-enabled-test-kserve-7dfddf4646-c84nz\" (UID: \"ee3e4b01-2183-44a8-8cc5-140f52eca09f\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-7dfddf4646-c84nz"
Jul 13 19:06:07.880488 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:06:07.880467    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee3e4b01-2183-44a8-8cc5-140f52eca09f-dshm\") pod \"auth-enabled-test-kserve-7dfddf4646-c84nz\" (UID: \"ee3e4b01-2183-44a8-8cc5-140f52eca09f\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-7dfddf4646-c84nz"
Jul 13 19:06:07.880748 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:06:07.880730    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee3e4b01-2183-44a8-8cc5-140f52eca09f-tls-certs\") pod \"auth-enabled-test-kserve-7dfddf4646-c84nz\" (UID: \"ee3e4b01-2183-44a8-8cc5-140f52eca09f\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-7dfddf4646-c84nz"
Jul 13 19:06:07.887601 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:06:07.887583    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4lnl\" (UniqueName: \"kubernetes.io/projected/ee3e4b01-2183-44a8-8cc5-140f52eca09f-kube-api-access-w4lnl\") pod \"auth-enabled-test-kserve-7dfddf4646-c84nz\" (UID: \"ee3e4b01-2183-44a8-8cc5-140f52eca09f\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-7dfddf4646-c84nz"
Jul 13 19:06:07.933526 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:06:07.933495    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-7b8b8fbfcb-h298k"]
Jul 13 19:06:07.936969 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:06:07.936952    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-7b8b8fbfcb-h298k"
Jul 13 19:06:07.939631 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:06:07.939610    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"auth-enabled-test-epp-sa-dockercfg-mnk2x\""
Jul 13 19:06:07.946897 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:06:07.946874    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-7b8b8fbfcb-h298k"]
Jul 13 19:06:08.006612 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:06:08.006586    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-7dfddf4646-c84nz"
Jul 13 19:06:08.049372 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:06:08.049340    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fc0c3d21-31f7-4b3f-aec7-576a5b9c5917" path="/var/lib/kubelet/pods/fc0c3d21-31f7-4b3f-aec7-576a5b9c5917/volumes"
Jul 13 19:06:08.080843 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:06:08.080812    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/670a8c6f-a148-48ab-b021-31d7aca0d8d9-kserve-provision-location\") pod \"auth-enabled-test-kserve-router-scheduler-7b8b8fbfcb-h298k\" (UID: \"670a8c6f-a148-48ab-b021-31d7aca0d8d9\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-7b8b8fbfcb-h298k"
Jul 13 19:06:08.080994 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:06:08.080886    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/670a8c6f-a148-48ab-b021-31d7aca0d8d9-tokenizer-uds\") pod \"auth-enabled-test-kserve-router-scheduler-7b8b8fbfcb-h298k\" (UID: \"670a8c6f-a148-48ab-b021-31d7aca0d8d9\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-7b8b8fbfcb-h298k"
Jul 13 19:06:08.080994 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:06:08.080945    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9nct\" (UniqueName: \"kubernetes.io/projected/670a8c6f-a148-48ab-b021-31d7aca0d8d9-kube-api-access-z9nct\") pod \"auth-enabled-test-kserve-router-scheduler-7b8b8fbfcb-h298k\" (UID: \"670a8c6f-a148-48ab-b021-31d7aca0d8d9\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-7b8b8fbfcb-h298k"
Jul 13 19:06:08.081115 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:06:08.081101    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/670a8c6f-a148-48ab-b021-31d7aca0d8d9-tls-certs\") pod \"auth-enabled-test-kserve-router-scheduler-7b8b8fbfcb-h298k\" (UID: \"670a8c6f-a148-48ab-b021-31d7aca0d8d9\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-7b8b8fbfcb-h298k"
Jul 13 19:06:08.142750 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:06:08.142685    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/auth-enabled-test-kserve-7dfddf4646-c84nz"]
Jul 13 19:06:08.146257 ip-10-0-128-215 kubenswrapper[2565]: W0713 19:06:08.146220    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee3e4b01_2183_44a8_8cc5_140f52eca09f.slice/crio-324b28392eb3c6755e4c43f779234035b713ed63cfeb6ec1c3c869359d3eba91 WatchSource:0}: Error finding container 324b28392eb3c6755e4c43f779234035b713ed63cfeb6ec1c3c869359d3eba91: Status 404 returned error can't find the container with id 324b28392eb3c6755e4c43f779234035b713ed63cfeb6ec1c3c869359d3eba91
Jul 13 19:06:08.181631 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:06:08.181607    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/670a8c6f-a148-48ab-b021-31d7aca0d8d9-tokenizer-uds\") pod \"auth-enabled-test-kserve-router-scheduler-7b8b8fbfcb-h298k\" (UID: \"670a8c6f-a148-48ab-b021-31d7aca0d8d9\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-7b8b8fbfcb-h298k"
Jul 13 19:06:08.181770 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:06:08.181656    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9nct\" (UniqueName: \"kubernetes.io/projected/670a8c6f-a148-48ab-b021-31d7aca0d8d9-kube-api-access-z9nct\") pod \"auth-enabled-test-kserve-router-scheduler-7b8b8fbfcb-h298k\" (UID: \"670a8c6f-a148-48ab-b021-31d7aca0d8d9\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-7b8b8fbfcb-h298k"
Jul 13 19:06:08.181851 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:06:08.181832    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/670a8c6f-a148-48ab-b021-31d7aca0d8d9-tls-certs\") pod \"auth-enabled-test-kserve-router-scheduler-7b8b8fbfcb-h298k\" (UID: \"670a8c6f-a148-48ab-b021-31d7aca0d8d9\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-7b8b8fbfcb-h298k"
Jul 13 19:06:08.181930 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:06:08.181888    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/670a8c6f-a148-48ab-b021-31d7aca0d8d9-kserve-provision-location\") pod \"auth-enabled-test-kserve-router-scheduler-7b8b8fbfcb-h298k\" (UID: \"670a8c6f-a148-48ab-b021-31d7aca0d8d9\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-7b8b8fbfcb-h298k"
Jul 13 19:06:08.182058 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:06:08.182041    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/670a8c6f-a148-48ab-b021-31d7aca0d8d9-tokenizer-uds\") pod \"auth-enabled-test-kserve-router-scheduler-7b8b8fbfcb-h298k\" (UID: \"670a8c6f-a148-48ab-b021-31d7aca0d8d9\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-7b8b8fbfcb-h298k"
Jul 13 19:06:08.182186 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:06:08.182170    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/670a8c6f-a148-48ab-b021-31d7aca0d8d9-kserve-provision-location\") pod \"auth-enabled-test-kserve-router-scheduler-7b8b8fbfcb-h298k\" (UID: \"670a8c6f-a148-48ab-b021-31d7aca0d8d9\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-7b8b8fbfcb-h298k"
Jul 13 19:06:08.184222 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:06:08.184198    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/670a8c6f-a148-48ab-b021-31d7aca0d8d9-tls-certs\") pod \"auth-enabled-test-kserve-router-scheduler-7b8b8fbfcb-h298k\" (UID: \"670a8c6f-a148-48ab-b021-31d7aca0d8d9\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-7b8b8fbfcb-h298k"
Jul 13 19:06:08.194672 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:06:08.194622    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9nct\" (UniqueName: \"kubernetes.io/projected/670a8c6f-a148-48ab-b021-31d7aca0d8d9-kube-api-access-z9nct\") pod \"auth-enabled-test-kserve-router-scheduler-7b8b8fbfcb-h298k\" (UID: \"670a8c6f-a148-48ab-b021-31d7aca0d8d9\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-7b8b8fbfcb-h298k"
Jul 13 19:06:08.247007 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:06:08.246975    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-7b8b8fbfcb-h298k"
Jul 13 19:06:08.386893 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:06:08.386866    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-7b8b8fbfcb-h298k"]
Jul 13 19:06:08.388730 ip-10-0-128-215 kubenswrapper[2565]: W0713 19:06:08.388680    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod670a8c6f_a148_48ab_b021_31d7aca0d8d9.slice/crio-5698296ca768e4eaf6c48c7047492a42ac6d41a137991131cde416c457c49cd1 WatchSource:0}: Error finding container 5698296ca768e4eaf6c48c7047492a42ac6d41a137991131cde416c457c49cd1: Status 404 returned error can't find the container with id 5698296ca768e4eaf6c48c7047492a42ac6d41a137991131cde416c457c49cd1
Jul 13 19:06:08.696107 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:06:08.696010    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-7b8b8fbfcb-h298k" event={"ID":"670a8c6f-a148-48ab-b021-31d7aca0d8d9","Type":"ContainerStarted","Data":"6d9d2afabe47ccf827f0ded3d7dc5c7889e488b575dbf425a3a88e03739fba51"}
Jul 13 19:06:08.696107 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:06:08.696071    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-7b8b8fbfcb-h298k" event={"ID":"670a8c6f-a148-48ab-b021-31d7aca0d8d9","Type":"ContainerStarted","Data":"5698296ca768e4eaf6c48c7047492a42ac6d41a137991131cde416c457c49cd1"}
Jul 13 19:06:08.697648 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:06:08.697624    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-7dfddf4646-c84nz" event={"ID":"ee3e4b01-2183-44a8-8cc5-140f52eca09f","Type":"ContainerStarted","Data":"c113c7dcdb607148addbff7f726ab1877f737808a04a188f86af8e12a5580cae"}
Jul 13 19:06:08.697719 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:06:08.697666    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-7dfddf4646-c84nz" event={"ID":"ee3e4b01-2183-44a8-8cc5-140f52eca09f","Type":"ContainerStarted","Data":"324b28392eb3c6755e4c43f779234035b713ed63cfeb6ec1c3c869359d3eba91"}
Jul 13 19:06:09.704232 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:06:09.704199    2565 generic.go:356] "Generic (PLEG): container finished" podID="670a8c6f-a148-48ab-b021-31d7aca0d8d9" containerID="6d9d2afabe47ccf827f0ded3d7dc5c7889e488b575dbf425a3a88e03739fba51" exitCode=0
Jul 13 19:06:09.704595 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:06:09.704279    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-7b8b8fbfcb-h298k" event={"ID":"670a8c6f-a148-48ab-b021-31d7aca0d8d9","Type":"ContainerDied","Data":"6d9d2afabe47ccf827f0ded3d7dc5c7889e488b575dbf425a3a88e03739fba51"}
Jul 13 19:06:11.720924 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:06:11.720820    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-7b8b8fbfcb-h298k" event={"ID":"670a8c6f-a148-48ab-b021-31d7aca0d8d9","Type":"ContainerStarted","Data":"651dc30cffa08b9a3cd2f39116be843ec6f1f311b02cf8fefd91cbedf3c343bc"}
Jul 13 19:06:11.721387 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:06:11.721053    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-7b8b8fbfcb-h298k"
Jul 13 19:06:11.722550 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:06:11.722528    2565 generic.go:356] "Generic (PLEG): container finished" podID="ee3e4b01-2183-44a8-8cc5-140f52eca09f" containerID="c113c7dcdb607148addbff7f726ab1877f737808a04a188f86af8e12a5580cae" exitCode=0
Jul 13 19:06:11.722660 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:06:11.722575    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-7dfddf4646-c84nz" event={"ID":"ee3e4b01-2183-44a8-8cc5-140f52eca09f","Type":"ContainerDied","Data":"c113c7dcdb607148addbff7f726ab1877f737808a04a188f86af8e12a5580cae"}
Jul 13 19:06:11.745868 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:06:11.745821    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-7b8b8fbfcb-h298k" podStartSLOduration=2.411921521 podStartE2EDuration="4.745806582s" totalImagesPullingTime="1.571698447s" totalInitContainerRuntime="762.186614ms" isStatefulPod=false podCreationTimestamp="2026-07-13 19:06:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 19:06:11.74300268 +0000 UTC m=+958.268136108" watchObservedRunningTime="2026-07-13 19:06:11.745806582 +0000 UTC m=+958.270939999"
Jul 13 19:06:26.698392 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:06:26.698356    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6ffc955574-p26g2"]
Jul 13 19:06:38.890931 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:06:38.890901    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-7dfddf4646-c84nz" event={"ID":"ee3e4b01-2183-44a8-8cc5-140f52eca09f","Type":"ContainerStarted","Data":"913eddd3672b76300f44d364ed7c7c97eb38746ea38ad022a4eab701b18a2497"}
Jul 13 19:06:39.924555 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:06:39.924502    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-7dfddf4646-c84nz" podStartSLOduration=3.022372962 podStartE2EDuration="32.924486108s" totalImagesPullingTime="27.07177035s" totalInitContainerRuntime="2.830342796s" isStatefulPod=false podCreationTimestamp="2026-07-13 19:06:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 19:06:39.92153769 +0000 UTC m=+986.446671117" watchObservedRunningTime="2026-07-13 19:06:39.924486108 +0000 UTC m=+986.449619523"
Jul 13 19:06:40.007733 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:06:40.007677    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-7dfddf4646-c84nz"
Jul 13 19:06:41.725913 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:06:41.725871    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6ffc955574-p26g2" podUID="6eaf9d6b-03bc-43c8-9d86-1a573a3d587b" containerName="main" containerID="cri-o://fba6c0583a498c972ce649519893874c83d21e957fbe5725a4bcbfb16d798108" gracePeriod=45
Jul 13 19:06:41.919002 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:06:41.918754    2565 generic.go:356] "Generic (PLEG): container finished" podID="6eaf9d6b-03bc-43c8-9d86-1a573a3d587b" containerID="fba6c0583a498c972ce649519893874c83d21e957fbe5725a4bcbfb16d798108" exitCode=0
Jul 13 19:06:41.919002 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:06:41.918920    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6ffc955574-p26g2" event={"ID":"6eaf9d6b-03bc-43c8-9d86-1a573a3d587b","Type":"ContainerDied","Data":"fba6c0583a498c972ce649519893874c83d21e957fbe5725a4bcbfb16d798108"}
Jul 13 19:06:41.985813 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:06:41.985756    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6ffc955574-p26g2"
Jul 13 19:06:42.144297 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:06:42.144260    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6eaf9d6b-03bc-43c8-9d86-1a573a3d587b-tls-certs\") pod \"6eaf9d6b-03bc-43c8-9d86-1a573a3d587b\" (UID: \"6eaf9d6b-03bc-43c8-9d86-1a573a3d587b\") "
Jul 13 19:06:42.144464 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:06:42.144343    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6eaf9d6b-03bc-43c8-9d86-1a573a3d587b-dshm\") pod \"6eaf9d6b-03bc-43c8-9d86-1a573a3d587b\" (UID: \"6eaf9d6b-03bc-43c8-9d86-1a573a3d587b\") "
Jul 13 19:06:42.144464 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:06:42.144435    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6eaf9d6b-03bc-43c8-9d86-1a573a3d587b-model-cache\") pod \"6eaf9d6b-03bc-43c8-9d86-1a573a3d587b\" (UID: \"6eaf9d6b-03bc-43c8-9d86-1a573a3d587b\") "
Jul 13 19:06:42.144600 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:06:42.144477    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6eaf9d6b-03bc-43c8-9d86-1a573a3d587b-home\") pod \"6eaf9d6b-03bc-43c8-9d86-1a573a3d587b\" (UID: \"6eaf9d6b-03bc-43c8-9d86-1a573a3d587b\") "
Jul 13 19:06:42.144600 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:06:42.144513    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-746rz\" (UniqueName: \"kubernetes.io/projected/6eaf9d6b-03bc-43c8-9d86-1a573a3d587b-kube-api-access-746rz\") pod \"6eaf9d6b-03bc-43c8-9d86-1a573a3d587b\" (UID: \"6eaf9d6b-03bc-43c8-9d86-1a573a3d587b\") "
Jul 13 19:06:42.144600 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:06:42.144530    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6eaf9d6b-03bc-43c8-9d86-1a573a3d587b-kserve-provision-location\") pod \"6eaf9d6b-03bc-43c8-9d86-1a573a3d587b\" (UID: \"6eaf9d6b-03bc-43c8-9d86-1a573a3d587b\") "
Jul 13 19:06:42.144600 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:06:42.144585    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6eaf9d6b-03bc-43c8-9d86-1a573a3d587b-tmp-dir\") pod \"6eaf9d6b-03bc-43c8-9d86-1a573a3d587b\" (UID: \"6eaf9d6b-03bc-43c8-9d86-1a573a3d587b\") "
Jul 13 19:06:42.144849 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:06:42.144727    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6eaf9d6b-03bc-43c8-9d86-1a573a3d587b-home" (OuterVolumeSpecName: "home") pod "6eaf9d6b-03bc-43c8-9d86-1a573a3d587b" (UID: "6eaf9d6b-03bc-43c8-9d86-1a573a3d587b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:06:42.144849 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:06:42.144738    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6eaf9d6b-03bc-43c8-9d86-1a573a3d587b-model-cache" (OuterVolumeSpecName: "model-cache") pod "6eaf9d6b-03bc-43c8-9d86-1a573a3d587b" (UID: "6eaf9d6b-03bc-43c8-9d86-1a573a3d587b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:06:42.144961 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:06:42.144935    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6eaf9d6b-03bc-43c8-9d86-1a573a3d587b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6eaf9d6b-03bc-43c8-9d86-1a573a3d587b" (UID: "6eaf9d6b-03bc-43c8-9d86-1a573a3d587b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:06:42.145595 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:06:42.145096    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6eaf9d6b-03bc-43c8-9d86-1a573a3d587b-model-cache\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 13 19:06:42.145595 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:06:42.145119    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6eaf9d6b-03bc-43c8-9d86-1a573a3d587b-home\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 13 19:06:42.145595 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:06:42.145132    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6eaf9d6b-03bc-43c8-9d86-1a573a3d587b-tmp-dir\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 13 19:06:42.147045 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:06:42.147009    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6eaf9d6b-03bc-43c8-9d86-1a573a3d587b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6eaf9d6b-03bc-43c8-9d86-1a573a3d587b" (UID: "6eaf9d6b-03bc-43c8-9d86-1a573a3d587b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 19:06:42.147148 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:06:42.147033    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6eaf9d6b-03bc-43c8-9d86-1a573a3d587b-kube-api-access-746rz" (OuterVolumeSpecName: "kube-api-access-746rz") pod "6eaf9d6b-03bc-43c8-9d86-1a573a3d587b" (UID: "6eaf9d6b-03bc-43c8-9d86-1a573a3d587b"). InnerVolumeSpecName "kube-api-access-746rz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 19:06:42.147306 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:06:42.147276    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6eaf9d6b-03bc-43c8-9d86-1a573a3d587b-dshm" (OuterVolumeSpecName: "dshm") pod "6eaf9d6b-03bc-43c8-9d86-1a573a3d587b" (UID: "6eaf9d6b-03bc-43c8-9d86-1a573a3d587b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:06:42.198677 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:06:42.198637    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6eaf9d6b-03bc-43c8-9d86-1a573a3d587b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6eaf9d6b-03bc-43c8-9d86-1a573a3d587b" (UID: "6eaf9d6b-03bc-43c8-9d86-1a573a3d587b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:06:42.246300 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:06:42.246230    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-746rz\" (UniqueName: \"kubernetes.io/projected/6eaf9d6b-03bc-43c8-9d86-1a573a3d587b-kube-api-access-746rz\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 13 19:06:42.246300 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:06:42.246254    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6eaf9d6b-03bc-43c8-9d86-1a573a3d587b-kserve-provision-location\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 13 19:06:42.246300 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:06:42.246266    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6eaf9d6b-03bc-43c8-9d86-1a573a3d587b-tls-certs\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 13 19:06:42.246300 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:06:42.246275    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6eaf9d6b-03bc-43c8-9d86-1a573a3d587b-dshm\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 13 19:06:42.732609 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:06:42.732582    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-7b8b8fbfcb-h298k"
Jul 13 19:06:42.926290 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:06:42.926262    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6ffc955574-p26g2"
Jul 13 19:06:42.926448 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:06:42.926261    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6ffc955574-p26g2" event={"ID":"6eaf9d6b-03bc-43c8-9d86-1a573a3d587b","Type":"ContainerDied","Data":"037fe3e87bf3f36fd2d31bb1568b1f376e06a2b1a496b8be4636b268b316cb7f"}
Jul 13 19:06:42.926448 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:06:42.926424    2565 scope.go:117] "RemoveContainer" containerID="fba6c0583a498c972ce649519893874c83d21e957fbe5725a4bcbfb16d798108"
Jul 13 19:06:42.937746 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:06:42.937727    2565 scope.go:117] "RemoveContainer" containerID="bfc248efa90a6dab0462762101a4d38c0620537c54ddbca0309b00f286c8c4b5"
Jul 13 19:06:42.949862 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:06:42.949836    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6ffc955574-p26g2"]
Jul 13 19:06:42.957251 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:06:42.957227    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6ffc955574-p26g2"]
Jul 13 19:06:44.048314 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:06:44.048285    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6eaf9d6b-03bc-43c8-9d86-1a573a3d587b" path="/var/lib/kubelet/pods/6eaf9d6b-03bc-43c8-9d86-1a573a3d587b/volumes"
Jul 13 19:06:48.007320 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:06:48.007279    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-7dfddf4646-c84nz"
Jul 13 19:06:48.009115 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:06:48.009085    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-7dfddf4646-c84nz" podUID="ee3e4b01-2183-44a8-8cc5-140f52eca09f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": dial tcp 10.134.0.41:8000: connect: connection refused"
Jul 13 19:06:58.007148 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:06:58.007105    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-7dfddf4646-c84nz" podUID="ee3e4b01-2183-44a8-8cc5-140f52eca09f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": dial tcp 10.134.0.41:8000: connect: connection refused"
Jul 13 19:07:08.007327 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:07:08.007276    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-7dfddf4646-c84nz" podUID="ee3e4b01-2183-44a8-8cc5-140f52eca09f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": dial tcp 10.134.0.41:8000: connect: connection refused"
Jul 13 19:07:18.008063 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:07:18.008018    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-7dfddf4646-c84nz" podUID="ee3e4b01-2183-44a8-8cc5-140f52eca09f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": dial tcp 10.134.0.41:8000: connect: connection refused"
Jul 13 19:07:28.007325 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:07:28.007274    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-7dfddf4646-c84nz" podUID="ee3e4b01-2183-44a8-8cc5-140f52eca09f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": dial tcp 10.134.0.41:8000: connect: connection refused"
Jul 13 19:07:38.007010 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:07:38.006967    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-7dfddf4646-c84nz" podUID="ee3e4b01-2183-44a8-8cc5-140f52eca09f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": dial tcp 10.134.0.41:8000: connect: connection refused"
Jul 13 19:07:48.007935 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:07:48.007895    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-7dfddf4646-c84nz" podUID="ee3e4b01-2183-44a8-8cc5-140f52eca09f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": dial tcp 10.134.0.41:8000: connect: connection refused"
Jul 13 19:07:58.007748 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:07:58.007666    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-7dfddf4646-c84nz" podUID="ee3e4b01-2183-44a8-8cc5-140f52eca09f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": dial tcp 10.134.0.41:8000: connect: connection refused"
Jul 13 19:08:08.007849 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:08:08.007747    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-7dfddf4646-c84nz" podUID="ee3e4b01-2183-44a8-8cc5-140f52eca09f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": dial tcp 10.134.0.41:8000: connect: connection refused"
Jul 13 19:08:18.007558 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:08:18.007506    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-7dfddf4646-c84nz" podUID="ee3e4b01-2183-44a8-8cc5-140f52eca09f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": dial tcp 10.134.0.41:8000: connect: connection refused"
Jul 13 19:08:28.017021 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:08:28.016993    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-7dfddf4646-c84nz"
Jul 13 19:08:28.024482 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:08:28.024461    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-7dfddf4646-c84nz"
Jul 13 19:08:32.362130 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:08:32.362089    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/auth-enabled-test-kserve-7dfddf4646-c84nz"]
Jul 13 19:08:32.370782 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:08:32.370755    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-7b8b8fbfcb-h298k"]
Jul 13 19:08:32.380134 ip-10-0-128-215 kubenswrapper[2565]: E0713 19:08:32.380092    2565 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 13 19:08:32.380134 ip-10-0-128-215 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 13 19:08:32.380134 ip-10-0-128-215 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Jul 13 19:08:32.380134 ip-10-0-128-215 kubenswrapper[2565]:  > containerID="651dc30cffa08b9a3cd2f39116be843ec6f1f311b02cf8fefd91cbedf3c343bc" cmd=["/bin/sleep","15"]
Jul 13 19:08:32.380435 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:08:32.380153    2565 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-7b8b8fbfcb-h298k" message="" err=<
Jul 13 19:08:32.380435 ip-10-0-128-215 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 13 19:08:32.380435 ip-10-0-128-215 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Jul 13 19:08:32.380435 ip-10-0-128-215 kubenswrapper[2565]:  >
Jul 13 19:08:32.380435 ip-10-0-128-215 kubenswrapper[2565]: E0713 19:08:32.380169    2565 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 13 19:08:32.380435 ip-10-0-128-215 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 13 19:08:32.380435 ip-10-0-128-215 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Jul 13 19:08:32.380435 ip-10-0-128-215 kubenswrapper[2565]:  > pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-7b8b8fbfcb-h298k" podUID="670a8c6f-a148-48ab-b021-31d7aca0d8d9" containerName="main" containerID="cri-o://651dc30cffa08b9a3cd2f39116be843ec6f1f311b02cf8fefd91cbedf3c343bc"
Jul 13 19:08:32.380435 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:08:32.380214    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-7b8b8fbfcb-h298k" podUID="670a8c6f-a148-48ab-b021-31d7aca0d8d9" containerName="main" containerID="cri-o://651dc30cffa08b9a3cd2f39116be843ec6f1f311b02cf8fefd91cbedf3c343bc" gracePeriod=60
Jul 13 19:08:32.513275 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:08:32.513247    2565 generic.go:356] "Generic (PLEG): container finished" podID="670a8c6f-a148-48ab-b021-31d7aca0d8d9" containerID="651dc30cffa08b9a3cd2f39116be843ec6f1f311b02cf8fefd91cbedf3c343bc" exitCode=0
Jul 13 19:08:32.513406 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:08:32.513316    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-7b8b8fbfcb-h298k" event={"ID":"670a8c6f-a148-48ab-b021-31d7aca0d8d9","Type":"ContainerDied","Data":"651dc30cffa08b9a3cd2f39116be843ec6f1f311b02cf8fefd91cbedf3c343bc"}
Jul 13 19:08:32.631588 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:08:32.631567    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-7b8b8fbfcb-h298k"
Jul 13 19:08:32.681231 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:08:32.681209    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/670a8c6f-a148-48ab-b021-31d7aca0d8d9-tokenizer-uds\") pod \"670a8c6f-a148-48ab-b021-31d7aca0d8d9\" (UID: \"670a8c6f-a148-48ab-b021-31d7aca0d8d9\") "
Jul 13 19:08:32.681382 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:08:32.681252    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z9nct\" (UniqueName: \"kubernetes.io/projected/670a8c6f-a148-48ab-b021-31d7aca0d8d9-kube-api-access-z9nct\") pod \"670a8c6f-a148-48ab-b021-31d7aca0d8d9\" (UID: \"670a8c6f-a148-48ab-b021-31d7aca0d8d9\") "
Jul 13 19:08:32.681382 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:08:32.681282    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/670a8c6f-a148-48ab-b021-31d7aca0d8d9-kserve-provision-location\") pod \"670a8c6f-a148-48ab-b021-31d7aca0d8d9\" (UID: \"670a8c6f-a148-48ab-b021-31d7aca0d8d9\") "
Jul 13 19:08:32.681382 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:08:32.681350    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/670a8c6f-a148-48ab-b021-31d7aca0d8d9-tls-certs\") pod \"670a8c6f-a148-48ab-b021-31d7aca0d8d9\" (UID: \"670a8c6f-a148-48ab-b021-31d7aca0d8d9\") "
Jul 13 19:08:32.681572 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:08:32.681523    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/670a8c6f-a148-48ab-b021-31d7aca0d8d9-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "670a8c6f-a148-48ab-b021-31d7aca0d8d9" (UID: "670a8c6f-a148-48ab-b021-31d7aca0d8d9"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:08:32.681661 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:08:32.681639    2565 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/670a8c6f-a148-48ab-b021-31d7aca0d8d9-tokenizer-uds\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 13 19:08:32.681986 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:08:32.681962    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/670a8c6f-a148-48ab-b021-31d7aca0d8d9-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "670a8c6f-a148-48ab-b021-31d7aca0d8d9" (UID: "670a8c6f-a148-48ab-b021-31d7aca0d8d9"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:08:32.683470 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:08:32.683449    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/670a8c6f-a148-48ab-b021-31d7aca0d8d9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "670a8c6f-a148-48ab-b021-31d7aca0d8d9" (UID: "670a8c6f-a148-48ab-b021-31d7aca0d8d9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 19:08:32.683470 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:08:32.683452    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/670a8c6f-a148-48ab-b021-31d7aca0d8d9-kube-api-access-z9nct" (OuterVolumeSpecName: "kube-api-access-z9nct") pod "670a8c6f-a148-48ab-b021-31d7aca0d8d9" (UID: "670a8c6f-a148-48ab-b021-31d7aca0d8d9"). InnerVolumeSpecName "kube-api-access-z9nct". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 19:08:32.782665 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:08:32.782639    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/670a8c6f-a148-48ab-b021-31d7aca0d8d9-tls-certs\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 13 19:08:32.782665 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:08:32.782661    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z9nct\" (UniqueName: \"kubernetes.io/projected/670a8c6f-a148-48ab-b021-31d7aca0d8d9-kube-api-access-z9nct\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 13 19:08:32.782665 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:08:32.782670    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/670a8c6f-a148-48ab-b021-31d7aca0d8d9-kserve-provision-location\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 13 19:08:33.520023 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:08:33.519988    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-7b8b8fbfcb-h298k"
Jul 13 19:08:33.520496 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:08:33.520020    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-7b8b8fbfcb-h298k" event={"ID":"670a8c6f-a148-48ab-b021-31d7aca0d8d9","Type":"ContainerDied","Data":"5698296ca768e4eaf6c48c7047492a42ac6d41a137991131cde416c457c49cd1"}
Jul 13 19:08:33.520496 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:08:33.520124    2565 scope.go:117] "RemoveContainer" containerID="651dc30cffa08b9a3cd2f39116be843ec6f1f311b02cf8fefd91cbedf3c343bc"
Jul 13 19:08:33.529601 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:08:33.529585    2565 scope.go:117] "RemoveContainer" containerID="6d9d2afabe47ccf827f0ded3d7dc5c7889e488b575dbf425a3a88e03739fba51"
Jul 13 19:08:33.552330 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:08:33.552309    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-7b8b8fbfcb-h298k"]
Jul 13 19:08:33.564333 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:08:33.564314    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-7b8b8fbfcb-h298k"]
Jul 13 19:08:34.047887 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:08:34.047855    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="670a8c6f-a148-48ab-b021-31d7aca0d8d9" path="/var/lib/kubelet/pods/670a8c6f-a148-48ab-b021-31d7aca0d8d9/volumes"
Jul 13 19:08:47.388173 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:08:47.388099    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-7dfddf4646-c84nz" podUID="ee3e4b01-2183-44a8-8cc5-140f52eca09f" containerName="main" containerID="cri-o://913eddd3672b76300f44d364ed7c7c97eb38746ea38ad022a4eab701b18a2497" gracePeriod=45
Jul 13 19:08:48.040184 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:08:48.040161    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-7dfddf4646-c84nz"
Jul 13 19:08:48.217199 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:08:48.217114    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w4lnl\" (UniqueName: \"kubernetes.io/projected/ee3e4b01-2183-44a8-8cc5-140f52eca09f-kube-api-access-w4lnl\") pod \"ee3e4b01-2183-44a8-8cc5-140f52eca09f\" (UID: \"ee3e4b01-2183-44a8-8cc5-140f52eca09f\") "
Jul 13 19:08:48.217199 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:08:48.217152    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee3e4b01-2183-44a8-8cc5-140f52eca09f-home\") pod \"ee3e4b01-2183-44a8-8cc5-140f52eca09f\" (UID: \"ee3e4b01-2183-44a8-8cc5-140f52eca09f\") "
Jul 13 19:08:48.217199 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:08:48.217181    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee3e4b01-2183-44a8-8cc5-140f52eca09f-tls-certs\") pod \"ee3e4b01-2183-44a8-8cc5-140f52eca09f\" (UID: \"ee3e4b01-2183-44a8-8cc5-140f52eca09f\") "
Jul 13 19:08:48.217472 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:08:48.217206    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee3e4b01-2183-44a8-8cc5-140f52eca09f-model-cache\") pod \"ee3e4b01-2183-44a8-8cc5-140f52eca09f\" (UID: \"ee3e4b01-2183-44a8-8cc5-140f52eca09f\") "
Jul 13 19:08:48.217472 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:08:48.217234    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ee3e4b01-2183-44a8-8cc5-140f52eca09f-kserve-provision-location\") pod \"ee3e4b01-2183-44a8-8cc5-140f52eca09f\" (UID: \"ee3e4b01-2183-44a8-8cc5-140f52eca09f\") "
Jul 13 19:08:48.217472 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:08:48.217319    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee3e4b01-2183-44a8-8cc5-140f52eca09f-tmp-dir\") pod \"ee3e4b01-2183-44a8-8cc5-140f52eca09f\" (UID: \"ee3e4b01-2183-44a8-8cc5-140f52eca09f\") "
Jul 13 19:08:48.217472 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:08:48.217347    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee3e4b01-2183-44a8-8cc5-140f52eca09f-dshm\") pod \"ee3e4b01-2183-44a8-8cc5-140f52eca09f\" (UID: \"ee3e4b01-2183-44a8-8cc5-140f52eca09f\") "
Jul 13 19:08:48.217660 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:08:48.217481    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee3e4b01-2183-44a8-8cc5-140f52eca09f-model-cache" (OuterVolumeSpecName: "model-cache") pod "ee3e4b01-2183-44a8-8cc5-140f52eca09f" (UID: "ee3e4b01-2183-44a8-8cc5-140f52eca09f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:08:48.217748 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:08:48.217682    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee3e4b01-2183-44a8-8cc5-140f52eca09f-model-cache\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 13 19:08:48.217878 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:08:48.217853    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee3e4b01-2183-44a8-8cc5-140f52eca09f-home" (OuterVolumeSpecName: "home") pod "ee3e4b01-2183-44a8-8cc5-140f52eca09f" (UID: "ee3e4b01-2183-44a8-8cc5-140f52eca09f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:08:48.219596 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:08:48.219573    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee3e4b01-2183-44a8-8cc5-140f52eca09f-dshm" (OuterVolumeSpecName: "dshm") pod "ee3e4b01-2183-44a8-8cc5-140f52eca09f" (UID: "ee3e4b01-2183-44a8-8cc5-140f52eca09f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:08:48.219942 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:08:48.219924    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee3e4b01-2183-44a8-8cc5-140f52eca09f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ee3e4b01-2183-44a8-8cc5-140f52eca09f" (UID: "ee3e4b01-2183-44a8-8cc5-140f52eca09f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 19:08:48.220026 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:08:48.219921    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee3e4b01-2183-44a8-8cc5-140f52eca09f-kube-api-access-w4lnl" (OuterVolumeSpecName: "kube-api-access-w4lnl") pod "ee3e4b01-2183-44a8-8cc5-140f52eca09f" (UID: "ee3e4b01-2183-44a8-8cc5-140f52eca09f"). InnerVolumeSpecName "kube-api-access-w4lnl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 19:08:48.235544 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:08:48.235513    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee3e4b01-2183-44a8-8cc5-140f52eca09f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ee3e4b01-2183-44a8-8cc5-140f52eca09f" (UID: "ee3e4b01-2183-44a8-8cc5-140f52eca09f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:08:48.278500 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:08:48.278474    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee3e4b01-2183-44a8-8cc5-140f52eca09f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ee3e4b01-2183-44a8-8cc5-140f52eca09f" (UID: "ee3e4b01-2183-44a8-8cc5-140f52eca09f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:08:48.318643 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:08:48.318621    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w4lnl\" (UniqueName: \"kubernetes.io/projected/ee3e4b01-2183-44a8-8cc5-140f52eca09f-kube-api-access-w4lnl\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 13 19:08:48.318643 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:08:48.318643    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee3e4b01-2183-44a8-8cc5-140f52eca09f-home\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 13 19:08:48.318777 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:08:48.318652    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee3e4b01-2183-44a8-8cc5-140f52eca09f-tls-certs\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 13 19:08:48.318777 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:08:48.318662    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ee3e4b01-2183-44a8-8cc5-140f52eca09f-kserve-provision-location\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 13 19:08:48.318777 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:08:48.318669    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee3e4b01-2183-44a8-8cc5-140f52eca09f-tmp-dir\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 13 19:08:48.318777 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:08:48.318677    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee3e4b01-2183-44a8-8cc5-140f52eca09f-dshm\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 13 19:08:48.593719 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:08:48.593673    2565 generic.go:356] "Generic (PLEG): container finished" podID="ee3e4b01-2183-44a8-8cc5-140f52eca09f" containerID="913eddd3672b76300f44d364ed7c7c97eb38746ea38ad022a4eab701b18a2497" exitCode=0
Jul 13 19:08:48.594180 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:08:48.593743    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-7dfddf4646-c84nz" event={"ID":"ee3e4b01-2183-44a8-8cc5-140f52eca09f","Type":"ContainerDied","Data":"913eddd3672b76300f44d364ed7c7c97eb38746ea38ad022a4eab701b18a2497"}
Jul 13 19:08:48.594180 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:08:48.593750    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-7dfddf4646-c84nz"
Jul 13 19:08:48.594180 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:08:48.593799    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-7dfddf4646-c84nz" event={"ID":"ee3e4b01-2183-44a8-8cc5-140f52eca09f","Type":"ContainerDied","Data":"324b28392eb3c6755e4c43f779234035b713ed63cfeb6ec1c3c869359d3eba91"}
Jul 13 19:08:48.594180 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:08:48.593830    2565 scope.go:117] "RemoveContainer" containerID="913eddd3672b76300f44d364ed7c7c97eb38746ea38ad022a4eab701b18a2497"
Jul 13 19:08:48.602940 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:08:48.602921    2565 scope.go:117] "RemoveContainer" containerID="c113c7dcdb607148addbff7f726ab1877f737808a04a188f86af8e12a5580cae"
Jul 13 19:08:48.610645 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:08:48.610628    2565 scope.go:117] "RemoveContainer" containerID="913eddd3672b76300f44d364ed7c7c97eb38746ea38ad022a4eab701b18a2497"
Jul 13 19:08:48.610937 ip-10-0-128-215 kubenswrapper[2565]: E0713 19:08:48.610908    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"913eddd3672b76300f44d364ed7c7c97eb38746ea38ad022a4eab701b18a2497\": container with ID starting with 913eddd3672b76300f44d364ed7c7c97eb38746ea38ad022a4eab701b18a2497 not found: ID does not exist" containerID="913eddd3672b76300f44d364ed7c7c97eb38746ea38ad022a4eab701b18a2497"
Jul 13 19:08:48.611003 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:08:48.610947    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"913eddd3672b76300f44d364ed7c7c97eb38746ea38ad022a4eab701b18a2497"} err="failed to get container status \"913eddd3672b76300f44d364ed7c7c97eb38746ea38ad022a4eab701b18a2497\": rpc error: code = NotFound desc = could not find container \"913eddd3672b76300f44d364ed7c7c97eb38746ea38ad022a4eab701b18a2497\": container with ID starting with 913eddd3672b76300f44d364ed7c7c97eb38746ea38ad022a4eab701b18a2497 not found: ID does not exist"
Jul 13 19:08:48.611003 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:08:48.610971    2565 scope.go:117] "RemoveContainer" containerID="c113c7dcdb607148addbff7f726ab1877f737808a04a188f86af8e12a5580cae"
Jul 13 19:08:48.611195 ip-10-0-128-215 kubenswrapper[2565]: E0713 19:08:48.611176    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c113c7dcdb607148addbff7f726ab1877f737808a04a188f86af8e12a5580cae\": container with ID starting with c113c7dcdb607148addbff7f726ab1877f737808a04a188f86af8e12a5580cae not found: ID does not exist" containerID="c113c7dcdb607148addbff7f726ab1877f737808a04a188f86af8e12a5580cae"
Jul 13 19:08:48.611254 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:08:48.611205    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c113c7dcdb607148addbff7f726ab1877f737808a04a188f86af8e12a5580cae"} err="failed to get container status \"c113c7dcdb607148addbff7f726ab1877f737808a04a188f86af8e12a5580cae\": rpc error: code = NotFound desc = could not find container \"c113c7dcdb607148addbff7f726ab1877f737808a04a188f86af8e12a5580cae\": container with ID starting with c113c7dcdb607148addbff7f726ab1877f737808a04a188f86af8e12a5580cae not found: ID does not exist"
Jul 13 19:08:48.621768 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:08:48.621746    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/auth-enabled-test-kserve-7dfddf4646-c84nz"]
Jul 13 19:08:48.626733 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:08:48.626687    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/auth-enabled-test-kserve-7dfddf4646-c84nz"]
Jul 13 19:08:49.018410 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:08:49.018376    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-7dfddf4646-c84nz" podUID="ee3e4b01-2183-44a8-8cc5-140f52eca09f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": context deadline exceeded"
Jul 13 19:08:50.047428 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:08:50.047386    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ee3e4b01-2183-44a8-8cc5-140f52eca09f" path="/var/lib/kubelet/pods/ee3e4b01-2183-44a8-8cc5-140f52eca09f/volumes"
Jul 13 19:09:03.391485 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:09:03.391451    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/auth-invalid-token-test-kserve-58b5cf756b-27d2s"]
Jul 13 19:09:03.392130 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:09:03.391838    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6eaf9d6b-03bc-43c8-9d86-1a573a3d587b" containerName="storage-initializer"
Jul 13 19:09:03.392130 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:09:03.391851    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="6eaf9d6b-03bc-43c8-9d86-1a573a3d587b" containerName="storage-initializer"
Jul 13 19:09:03.392130 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:09:03.391866    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="670a8c6f-a148-48ab-b021-31d7aca0d8d9" containerName="main"
Jul 13 19:09:03.392130 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:09:03.391871    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="670a8c6f-a148-48ab-b021-31d7aca0d8d9" containerName="main"
Jul 13 19:09:03.392130 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:09:03.391878    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="670a8c6f-a148-48ab-b021-31d7aca0d8d9" containerName="storage-initializer"
Jul 13 19:09:03.392130 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:09:03.391884    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="670a8c6f-a148-48ab-b021-31d7aca0d8d9" containerName="storage-initializer"
Jul 13 19:09:03.392130 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:09:03.391890    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee3e4b01-2183-44a8-8cc5-140f52eca09f" containerName="main"
Jul 13 19:09:03.392130 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:09:03.391895    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee3e4b01-2183-44a8-8cc5-140f52eca09f" containerName="main"
Jul 13 19:09:03.392130 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:09:03.391904    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee3e4b01-2183-44a8-8cc5-140f52eca09f" containerName="storage-initializer"
Jul 13 19:09:03.392130 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:09:03.391909    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee3e4b01-2183-44a8-8cc5-140f52eca09f" containerName="storage-initializer"
Jul 13 19:09:03.392130 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:09:03.391915    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6eaf9d6b-03bc-43c8-9d86-1a573a3d587b" containerName="main"
Jul 13 19:09:03.392130 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:09:03.391921    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="6eaf9d6b-03bc-43c8-9d86-1a573a3d587b" containerName="main"
Jul 13 19:09:03.392130 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:09:03.391979    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="ee3e4b01-2183-44a8-8cc5-140f52eca09f" containerName="main"
Jul 13 19:09:03.392130 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:09:03.391990    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="6eaf9d6b-03bc-43c8-9d86-1a573a3d587b" containerName="main"
Jul 13 19:09:03.392130 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:09:03.391998    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="670a8c6f-a148-48ab-b021-31d7aca0d8d9" containerName="main"
Jul 13 19:09:03.394964 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:09:03.394942    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-58b5cf756b-27d2s"
Jul 13 19:09:03.398040 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:09:03.398017    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 13 19:09:03.398645 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:09:03.398627    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-gv2sm\""
Jul 13 19:09:03.398747 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:09:03.398644    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 13 19:09:03.398747 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:09:03.398644    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 13 19:09:03.399291 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:09:03.399278    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"auth-invalid-token-test-kserve-self-signed-certs\""
Jul 13 19:09:03.415069 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:09:03.415041    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/auth-invalid-token-test-kserve-58b5cf756b-27d2s"]
Jul 13 19:09:03.444802 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:09:03.444770    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5261b96e-7755-486b-89fe-7875fbfe692e-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-58b5cf756b-27d2s\" (UID: \"5261b96e-7755-486b-89fe-7875fbfe692e\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-58b5cf756b-27d2s"
Jul 13 19:09:03.444802 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:09:03.444803    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5261b96e-7755-486b-89fe-7875fbfe692e-tls-certs\") pod \"auth-invalid-token-test-kserve-58b5cf756b-27d2s\" (UID: \"5261b96e-7755-486b-89fe-7875fbfe692e\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-58b5cf756b-27d2s"
Jul 13 19:09:03.445008 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:09:03.444886    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5261b96e-7755-486b-89fe-7875fbfe692e-model-cache\") pod \"auth-invalid-token-test-kserve-58b5cf756b-27d2s\" (UID: \"5261b96e-7755-486b-89fe-7875fbfe692e\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-58b5cf756b-27d2s"
Jul 13 19:09:03.445008 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:09:03.444902    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtzng\" (UniqueName: \"kubernetes.io/projected/5261b96e-7755-486b-89fe-7875fbfe692e-kube-api-access-qtzng\") pod \"auth-invalid-token-test-kserve-58b5cf756b-27d2s\" (UID: \"5261b96e-7755-486b-89fe-7875fbfe692e\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-58b5cf756b-27d2s"
Jul 13 19:09:03.445008 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:09:03.444920    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5261b96e-7755-486b-89fe-7875fbfe692e-dshm\") pod \"auth-invalid-token-test-kserve-58b5cf756b-27d2s\" (UID: \"5261b96e-7755-486b-89fe-7875fbfe692e\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-58b5cf756b-27d2s"
Jul 13 19:09:03.445008 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:09:03.444938    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5261b96e-7755-486b-89fe-7875fbfe692e-tmp-dir\") pod \"auth-invalid-token-test-kserve-58b5cf756b-27d2s\" (UID: \"5261b96e-7755-486b-89fe-7875fbfe692e\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-58b5cf756b-27d2s"
Jul 13 19:09:03.445008 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:09:03.444982    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5261b96e-7755-486b-89fe-7875fbfe692e-home\") pod \"auth-invalid-token-test-kserve-58b5cf756b-27d2s\" (UID: \"5261b96e-7755-486b-89fe-7875fbfe692e\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-58b5cf756b-27d2s"
Jul 13 19:09:03.546141 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:09:03.546103    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5261b96e-7755-486b-89fe-7875fbfe692e-model-cache\") pod \"auth-invalid-token-test-kserve-58b5cf756b-27d2s\" (UID: \"5261b96e-7755-486b-89fe-7875fbfe692e\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-58b5cf756b-27d2s"
Jul 13 19:09:03.546141 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:09:03.546140    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtzng\" (UniqueName: \"kubernetes.io/projected/5261b96e-7755-486b-89fe-7875fbfe692e-kube-api-access-qtzng\") pod \"auth-invalid-token-test-kserve-58b5cf756b-27d2s\" (UID: \"5261b96e-7755-486b-89fe-7875fbfe692e\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-58b5cf756b-27d2s"
Jul 13 19:09:03.546380 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:09:03.546165    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5261b96e-7755-486b-89fe-7875fbfe692e-dshm\") pod \"auth-invalid-token-test-kserve-58b5cf756b-27d2s\" (UID: \"5261b96e-7755-486b-89fe-7875fbfe692e\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-58b5cf756b-27d2s"
Jul 13 19:09:03.546380 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:09:03.546191    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5261b96e-7755-486b-89fe-7875fbfe692e-tmp-dir\") pod \"auth-invalid-token-test-kserve-58b5cf756b-27d2s\" (UID: \"5261b96e-7755-486b-89fe-7875fbfe692e\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-58b5cf756b-27d2s"
Jul 13 19:09:03.546380 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:09:03.546222    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5261b96e-7755-486b-89fe-7875fbfe692e-home\") pod \"auth-invalid-token-test-kserve-58b5cf756b-27d2s\" (UID: \"5261b96e-7755-486b-89fe-7875fbfe692e\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-58b5cf756b-27d2s"
Jul 13 19:09:03.546380 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:09:03.546295    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5261b96e-7755-486b-89fe-7875fbfe692e-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-58b5cf756b-27d2s\" (UID: \"5261b96e-7755-486b-89fe-7875fbfe692e\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-58b5cf756b-27d2s"
Jul 13 19:09:03.546380 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:09:03.546323    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5261b96e-7755-486b-89fe-7875fbfe692e-tls-certs\") pod \"auth-invalid-token-test-kserve-58b5cf756b-27d2s\" (UID: \"5261b96e-7755-486b-89fe-7875fbfe692e\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-58b5cf756b-27d2s"
Jul 13 19:09:03.546661 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:09:03.546593    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5261b96e-7755-486b-89fe-7875fbfe692e-model-cache\") pod \"auth-invalid-token-test-kserve-58b5cf756b-27d2s\" (UID: \"5261b96e-7755-486b-89fe-7875fbfe692e\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-58b5cf756b-27d2s"
Jul 13 19:09:03.546763 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:09:03.546662    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5261b96e-7755-486b-89fe-7875fbfe692e-home\") pod \"auth-invalid-token-test-kserve-58b5cf756b-27d2s\" (UID: \"5261b96e-7755-486b-89fe-7875fbfe692e\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-58b5cf756b-27d2s"
Jul 13 19:09:03.546824 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:09:03.546758    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5261b96e-7755-486b-89fe-7875fbfe692e-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-58b5cf756b-27d2s\" (UID: \"5261b96e-7755-486b-89fe-7875fbfe692e\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-58b5cf756b-27d2s"
Jul 13 19:09:03.546824 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:09:03.546787    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5261b96e-7755-486b-89fe-7875fbfe692e-tmp-dir\") pod \"auth-invalid-token-test-kserve-58b5cf756b-27d2s\" (UID: \"5261b96e-7755-486b-89fe-7875fbfe692e\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-58b5cf756b-27d2s"
Jul 13 19:09:03.548677 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:09:03.548657    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5261b96e-7755-486b-89fe-7875fbfe692e-dshm\") pod \"auth-invalid-token-test-kserve-58b5cf756b-27d2s\" (UID: \"5261b96e-7755-486b-89fe-7875fbfe692e\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-58b5cf756b-27d2s"
Jul 13 19:09:03.548992 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:09:03.548974    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5261b96e-7755-486b-89fe-7875fbfe692e-tls-certs\") pod \"auth-invalid-token-test-kserve-58b5cf756b-27d2s\" (UID: \"5261b96e-7755-486b-89fe-7875fbfe692e\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-58b5cf756b-27d2s"
Jul 13 19:09:03.554993 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:09:03.554971    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtzng\" (UniqueName: \"kubernetes.io/projected/5261b96e-7755-486b-89fe-7875fbfe692e-kube-api-access-qtzng\") pod \"auth-invalid-token-test-kserve-58b5cf756b-27d2s\" (UID: \"5261b96e-7755-486b-89fe-7875fbfe692e\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-58b5cf756b-27d2s"
Jul 13 19:09:03.705363 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:09:03.705260    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-58b5cf756b-27d2s"
Jul 13 19:09:03.721733 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:09:03.721688    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-6c486c59585thrg"]
Jul 13 19:09:03.725317 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:09:03.724550    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-6c486c59585thrg"
Jul 13 19:09:03.727637 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:09:03.727611    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"auth-invalid-token-test-epp-sa-dockercfg-kbz7f\""
Jul 13 19:09:03.737913 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:09:03.737888    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-6c486c59585thrg"]
Jul 13 19:09:03.747828 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:09:03.747801    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/4f31d263-8599-4d5b-8100-924cc7fdc7e9-tokenizer-uds\") pod \"auth-invalid-token-test-kserve-router-scheduler-6c486c59585thrg\" (UID: \"4f31d263-8599-4d5b-8100-924cc7fdc7e9\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-6c486c59585thrg"
Jul 13 19:09:03.747954 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:09:03.747841    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4f31d263-8599-4d5b-8100-924cc7fdc7e9-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-router-scheduler-6c486c59585thrg\" (UID: \"4f31d263-8599-4d5b-8100-924cc7fdc7e9\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-6c486c59585thrg"
Jul 13 19:09:03.747954 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:09:03.747881    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f31d263-8599-4d5b-8100-924cc7fdc7e9-tls-certs\") pod \"auth-invalid-token-test-kserve-router-scheduler-6c486c59585thrg\" (UID: \"4f31d263-8599-4d5b-8100-924cc7fdc7e9\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-6c486c59585thrg"
Jul 13 19:09:03.747954 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:09:03.747918    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpfcl\" (UniqueName: \"kubernetes.io/projected/4f31d263-8599-4d5b-8100-924cc7fdc7e9-kube-api-access-qpfcl\") pod \"auth-invalid-token-test-kserve-router-scheduler-6c486c59585thrg\" (UID: \"4f31d263-8599-4d5b-8100-924cc7fdc7e9\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-6c486c59585thrg"
Jul 13 19:09:03.848541 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:09:03.848510    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4f31d263-8599-4d5b-8100-924cc7fdc7e9-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-router-scheduler-6c486c59585thrg\" (UID: \"4f31d263-8599-4d5b-8100-924cc7fdc7e9\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-6c486c59585thrg"
Jul 13 19:09:03.848719 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:09:03.848567    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f31d263-8599-4d5b-8100-924cc7fdc7e9-tls-certs\") pod \"auth-invalid-token-test-kserve-router-scheduler-6c486c59585thrg\" (UID: \"4f31d263-8599-4d5b-8100-924cc7fdc7e9\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-6c486c59585thrg"
Jul 13 19:09:03.848719 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:09:03.848590    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpfcl\" (UniqueName: \"kubernetes.io/projected/4f31d263-8599-4d5b-8100-924cc7fdc7e9-kube-api-access-qpfcl\") pod \"auth-invalid-token-test-kserve-router-scheduler-6c486c59585thrg\" (UID: \"4f31d263-8599-4d5b-8100-924cc7fdc7e9\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-6c486c59585thrg"
Jul 13 19:09:03.848719 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:09:03.848650    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/4f31d263-8599-4d5b-8100-924cc7fdc7e9-tokenizer-uds\") pod \"auth-invalid-token-test-kserve-router-scheduler-6c486c59585thrg\" (UID: \"4f31d263-8599-4d5b-8100-924cc7fdc7e9\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-6c486c59585thrg"
Jul 13 19:09:03.848896 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:09:03.848877    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4f31d263-8599-4d5b-8100-924cc7fdc7e9-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-router-scheduler-6c486c59585thrg\" (UID: \"4f31d263-8599-4d5b-8100-924cc7fdc7e9\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-6c486c59585thrg"
Jul 13 19:09:03.848934 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:09:03.848920    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/4f31d263-8599-4d5b-8100-924cc7fdc7e9-tokenizer-uds\") pod \"auth-invalid-token-test-kserve-router-scheduler-6c486c59585thrg\" (UID: \"4f31d263-8599-4d5b-8100-924cc7fdc7e9\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-6c486c59585thrg"
Jul 13 19:09:03.851195 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:09:03.851167    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f31d263-8599-4d5b-8100-924cc7fdc7e9-tls-certs\") pod \"auth-invalid-token-test-kserve-router-scheduler-6c486c59585thrg\" (UID: \"4f31d263-8599-4d5b-8100-924cc7fdc7e9\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-6c486c59585thrg"
Jul 13 19:09:03.860557 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:09:03.860534    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpfcl\" (UniqueName: \"kubernetes.io/projected/4f31d263-8599-4d5b-8100-924cc7fdc7e9-kube-api-access-qpfcl\") pod \"auth-invalid-token-test-kserve-router-scheduler-6c486c59585thrg\" (UID: \"4f31d263-8599-4d5b-8100-924cc7fdc7e9\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-6c486c59585thrg"
Jul 13 19:09:03.865934 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:09:03.865915    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/auth-invalid-token-test-kserve-58b5cf756b-27d2s"]
Jul 13 19:09:03.867607 ip-10-0-128-215 kubenswrapper[2565]: W0713 19:09:03.867582    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5261b96e_7755_486b_89fe_7875fbfe692e.slice/crio-34f1ae3dc98a6855273cf144c2d25ab4d731199cf1d353fd981d76576f99fc03 WatchSource:0}: Error finding container 34f1ae3dc98a6855273cf144c2d25ab4d731199cf1d353fd981d76576f99fc03: Status 404 returned error can't find the container with id 34f1ae3dc98a6855273cf144c2d25ab4d731199cf1d353fd981d76576f99fc03
Jul 13 19:09:04.042587 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:09:04.042549    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-6c486c59585thrg"
Jul 13 19:09:04.186817 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:09:04.186790    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-6c486c59585thrg"]
Jul 13 19:09:04.188910 ip-10-0-128-215 kubenswrapper[2565]: W0713 19:09:04.188884    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f31d263_8599_4d5b_8100_924cc7fdc7e9.slice/crio-6c24314aad24dd5536da842d8a5be6cbae44d4a4a8cd8fc3613b96655aedffd5 WatchSource:0}: Error finding container 6c24314aad24dd5536da842d8a5be6cbae44d4a4a8cd8fc3613b96655aedffd5: Status 404 returned error can't find the container with id 6c24314aad24dd5536da842d8a5be6cbae44d4a4a8cd8fc3613b96655aedffd5
Jul 13 19:09:04.680491 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:09:04.680452    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-58b5cf756b-27d2s" event={"ID":"5261b96e-7755-486b-89fe-7875fbfe692e","Type":"ContainerStarted","Data":"f965e6feda3bc8f6eb07a71fce3261ed485c6f27ed17b05bd8ae99db552d6dff"}
Jul 13 19:09:04.680993 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:09:04.680520    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-58b5cf756b-27d2s" event={"ID":"5261b96e-7755-486b-89fe-7875fbfe692e","Type":"ContainerStarted","Data":"34f1ae3dc98a6855273cf144c2d25ab4d731199cf1d353fd981d76576f99fc03"}
Jul 13 19:09:04.682745 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:09:04.682686    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-6c486c59585thrg" event={"ID":"4f31d263-8599-4d5b-8100-924cc7fdc7e9","Type":"ContainerStarted","Data":"6a749e0c431741665213264eec85e8c820e24e8a3f8c641fcbea7f693aece293"}
Jul 13 19:09:04.682885 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:09:04.682763    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-6c486c59585thrg" event={"ID":"4f31d263-8599-4d5b-8100-924cc7fdc7e9","Type":"ContainerStarted","Data":"6c24314aad24dd5536da842d8a5be6cbae44d4a4a8cd8fc3613b96655aedffd5"}
Jul 13 19:09:05.695008 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:09:05.694972    2565 generic.go:356] "Generic (PLEG): container finished" podID="4f31d263-8599-4d5b-8100-924cc7fdc7e9" containerID="6a749e0c431741665213264eec85e8c820e24e8a3f8c641fcbea7f693aece293" exitCode=0
Jul 13 19:09:05.695404 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:09:05.695013    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-6c486c59585thrg" event={"ID":"4f31d263-8599-4d5b-8100-924cc7fdc7e9","Type":"ContainerDied","Data":"6a749e0c431741665213264eec85e8c820e24e8a3f8c641fcbea7f693aece293"}
Jul 13 19:09:06.701992 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:09:06.701951    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-6c486c59585thrg" event={"ID":"4f31d263-8599-4d5b-8100-924cc7fdc7e9","Type":"ContainerStarted","Data":"d98719d6208761166bc9c089a8be18e8e12e92cdfe3d47278ea8a97356ebcc22"}
Jul 13 19:09:06.702418 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:09:06.702093    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-6c486c59585thrg"
Jul 13 19:09:06.727280 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:09:06.727184    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-6c486c59585thrg" podStartSLOduration=2.940203917 podStartE2EDuration="3.727164202s" totalImagesPullingTime="0s" totalInitContainerRuntime="786.960285ms" isStatefulPod=false podCreationTimestamp="2026-07-13 19:09:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 19:09:06.723193183 +0000 UTC m=+1133.248326623" watchObservedRunningTime="2026-07-13 19:09:06.727164202 +0000 UTC m=+1133.252297621"
Jul 13 19:09:07.708772 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:09:07.708739    2565 generic.go:356] "Generic (PLEG): container finished" podID="5261b96e-7755-486b-89fe-7875fbfe692e" containerID="f965e6feda3bc8f6eb07a71fce3261ed485c6f27ed17b05bd8ae99db552d6dff" exitCode=0
Jul 13 19:09:07.709148 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:09:07.708813    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-58b5cf756b-27d2s" event={"ID":"5261b96e-7755-486b-89fe-7875fbfe692e","Type":"ContainerDied","Data":"f965e6feda3bc8f6eb07a71fce3261ed485c6f27ed17b05bd8ae99db552d6dff"}
Jul 13 19:09:08.717154 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:09:08.717118    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-58b5cf756b-27d2s" event={"ID":"5261b96e-7755-486b-89fe-7875fbfe692e","Type":"ContainerStarted","Data":"ca3cfb652d9117652936e007797aa21e9266b521d220e0927c3743ac4844d2f3"}
Jul 13 19:09:08.744531 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:09:08.744474    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-58b5cf756b-27d2s" podStartSLOduration=2.674593557 podStartE2EDuration="5.744457807s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.06986425s" isStatefulPod=false podCreationTimestamp="2026-07-13 19:09:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 19:09:08.741895048 +0000 UTC m=+1135.267028479" watchObservedRunningTime="2026-07-13 19:09:08.744457807 +0000 UTC m=+1135.269591222"
Jul 13 19:09:09.705419 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:09:09.705381    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-58b5cf756b-27d2s"
Jul 13 19:09:13.705567 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:09:13.705529    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-58b5cf756b-27d2s"
Jul 13 19:09:13.707389 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:09:13.707360    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-58b5cf756b-27d2s" podUID="5261b96e-7755-486b-89fe-7875fbfe692e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/health\": dial tcp 10.134.0.43:8000: connect: connection refused"
Jul 13 19:09:23.706120 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:09:23.706064    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-58b5cf756b-27d2s" podUID="5261b96e-7755-486b-89fe-7875fbfe692e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/health\": dial tcp 10.134.0.43:8000: connect: connection refused"
Jul 13 19:09:33.706390 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:09:33.706291    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-58b5cf756b-27d2s" podUID="5261b96e-7755-486b-89fe-7875fbfe692e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/health\": dial tcp 10.134.0.43:8000: connect: connection refused"
Jul 13 19:09:37.710721 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:09:37.710678    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-6c486c59585thrg"
Jul 13 19:09:43.705995 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:09:43.705946    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-58b5cf756b-27d2s" podUID="5261b96e-7755-486b-89fe-7875fbfe692e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/health\": dial tcp 10.134.0.43:8000: connect: connection refused"
Jul 13 19:09:53.705675 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:09:53.705629    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-58b5cf756b-27d2s" podUID="5261b96e-7755-486b-89fe-7875fbfe692e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/health\": dial tcp 10.134.0.43:8000: connect: connection refused"
Jul 13 19:10:03.705981 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:10:03.705936    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-58b5cf756b-27d2s" podUID="5261b96e-7755-486b-89fe-7875fbfe692e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/health\": dial tcp 10.134.0.43:8000: connect: connection refused"
Jul 13 19:10:13.705969 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:10:13.705918    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-58b5cf756b-27d2s" podUID="5261b96e-7755-486b-89fe-7875fbfe692e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/health\": dial tcp 10.134.0.43:8000: connect: connection refused"
Jul 13 19:10:13.951047 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:10:13.951017    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2bscc_2f0655bd-4a09-4314-8c1b-e30191eaec0e/ovn-acl-logging/0.log"
Jul 13 19:10:13.951391 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:10:13.951370    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2bscc_2f0655bd-4a09-4314-8c1b-e30191eaec0e/ovn-acl-logging/0.log"
Jul 13 19:10:23.706543 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:10:23.706493    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-58b5cf756b-27d2s" podUID="5261b96e-7755-486b-89fe-7875fbfe692e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/health\": dial tcp 10.134.0.43:8000: connect: connection refused"
Jul 13 19:10:33.705903 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:10:33.705857    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-58b5cf756b-27d2s" podUID="5261b96e-7755-486b-89fe-7875fbfe692e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/health\": dial tcp 10.134.0.43:8000: connect: connection refused"
Jul 13 19:10:43.706249 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:10:43.706202    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-58b5cf756b-27d2s" podUID="5261b96e-7755-486b-89fe-7875fbfe692e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/health\": dial tcp 10.134.0.43:8000: connect: connection refused"
Jul 13 19:10:53.706175 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:10:53.706129    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-58b5cf756b-27d2s" podUID="5261b96e-7755-486b-89fe-7875fbfe692e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/health\": dial tcp 10.134.0.43:8000: connect: connection refused"
Jul 13 19:11:03.715890 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:11:03.715805    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-58b5cf756b-27d2s"
Jul 13 19:11:03.723560 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:11:03.723534    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-58b5cf756b-27d2s"
Jul 13 19:11:05.549214 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:11:05.549174    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/auth-invalid-token-test-kserve-58b5cf756b-27d2s"]
Jul 13 19:11:05.573593 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:11:05.573563    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-6c486c59585thrg"]
Jul 13 19:11:05.583670 ip-10-0-128-215 kubenswrapper[2565]: E0713 19:11:05.583629    2565 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 13 19:11:05.583670 ip-10-0-128-215 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 13 19:11:05.583670 ip-10-0-128-215 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Jul 13 19:11:05.583670 ip-10-0-128-215 kubenswrapper[2565]:  > containerID="d98719d6208761166bc9c089a8be18e8e12e92cdfe3d47278ea8a97356ebcc22" cmd=["/bin/sleep","15"]
Jul 13 19:11:05.584040 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:11:05.583691    2565 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-6c486c59585thrg" message="" err=<
Jul 13 19:11:05.584040 ip-10-0-128-215 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 13 19:11:05.584040 ip-10-0-128-215 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Jul 13 19:11:05.584040 ip-10-0-128-215 kubenswrapper[2565]:  >
Jul 13 19:11:05.584040 ip-10-0-128-215 kubenswrapper[2565]: E0713 19:11:05.583723    2565 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 13 19:11:05.584040 ip-10-0-128-215 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 13 19:11:05.584040 ip-10-0-128-215 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Jul 13 19:11:05.584040 ip-10-0-128-215 kubenswrapper[2565]:  > pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-6c486c59585thrg" podUID="4f31d263-8599-4d5b-8100-924cc7fdc7e9" containerName="main" containerID="cri-o://d98719d6208761166bc9c089a8be18e8e12e92cdfe3d47278ea8a97356ebcc22"
Jul 13 19:11:05.584040 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:11:05.583766    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-6c486c59585thrg" podUID="4f31d263-8599-4d5b-8100-924cc7fdc7e9" containerName="main" containerID="cri-o://d98719d6208761166bc9c089a8be18e8e12e92cdfe3d47278ea8a97356ebcc22" gracePeriod=60
Jul 13 19:11:05.831721 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:11:05.831676    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-6c486c59585thrg"
Jul 13 19:11:05.979181 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:11:05.979156    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f31d263-8599-4d5b-8100-924cc7fdc7e9-tls-certs\") pod \"4f31d263-8599-4d5b-8100-924cc7fdc7e9\" (UID: \"4f31d263-8599-4d5b-8100-924cc7fdc7e9\") "
Jul 13 19:11:05.979326 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:11:05.979211    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qpfcl\" (UniqueName: \"kubernetes.io/projected/4f31d263-8599-4d5b-8100-924cc7fdc7e9-kube-api-access-qpfcl\") pod \"4f31d263-8599-4d5b-8100-924cc7fdc7e9\" (UID: \"4f31d263-8599-4d5b-8100-924cc7fdc7e9\") "
Jul 13 19:11:05.979326 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:11:05.979239    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/4f31d263-8599-4d5b-8100-924cc7fdc7e9-tokenizer-uds\") pod \"4f31d263-8599-4d5b-8100-924cc7fdc7e9\" (UID: \"4f31d263-8599-4d5b-8100-924cc7fdc7e9\") "
Jul 13 19:11:05.979326 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:11:05.979256    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4f31d263-8599-4d5b-8100-924cc7fdc7e9-kserve-provision-location\") pod \"4f31d263-8599-4d5b-8100-924cc7fdc7e9\" (UID: \"4f31d263-8599-4d5b-8100-924cc7fdc7e9\") "
Jul 13 19:11:05.979542 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:11:05.979506    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f31d263-8599-4d5b-8100-924cc7fdc7e9-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "4f31d263-8599-4d5b-8100-924cc7fdc7e9" (UID: "4f31d263-8599-4d5b-8100-924cc7fdc7e9"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:11:05.979983 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:11:05.979963    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f31d263-8599-4d5b-8100-924cc7fdc7e9-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4f31d263-8599-4d5b-8100-924cc7fdc7e9" (UID: "4f31d263-8599-4d5b-8100-924cc7fdc7e9"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:11:05.981437 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:11:05.981414    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f31d263-8599-4d5b-8100-924cc7fdc7e9-kube-api-access-qpfcl" (OuterVolumeSpecName: "kube-api-access-qpfcl") pod "4f31d263-8599-4d5b-8100-924cc7fdc7e9" (UID: "4f31d263-8599-4d5b-8100-924cc7fdc7e9"). InnerVolumeSpecName "kube-api-access-qpfcl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 19:11:05.981514 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:11:05.981427    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f31d263-8599-4d5b-8100-924cc7fdc7e9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4f31d263-8599-4d5b-8100-924cc7fdc7e9" (UID: "4f31d263-8599-4d5b-8100-924cc7fdc7e9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 19:11:06.080440 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:11:06.080369    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qpfcl\" (UniqueName: \"kubernetes.io/projected/4f31d263-8599-4d5b-8100-924cc7fdc7e9-kube-api-access-qpfcl\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 13 19:11:06.080440 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:11:06.080398    2565 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/4f31d263-8599-4d5b-8100-924cc7fdc7e9-tokenizer-uds\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 13 19:11:06.080440 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:11:06.080413    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4f31d263-8599-4d5b-8100-924cc7fdc7e9-kserve-provision-location\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 13 19:11:06.080440 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:11:06.080425    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f31d263-8599-4d5b-8100-924cc7fdc7e9-tls-certs\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 13 19:11:06.352639 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:11:06.352547    2565 generic.go:356] "Generic (PLEG): container finished" podID="4f31d263-8599-4d5b-8100-924cc7fdc7e9" containerID="d98719d6208761166bc9c089a8be18e8e12e92cdfe3d47278ea8a97356ebcc22" exitCode=0
Jul 13 19:11:06.352639 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:11:06.352596    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-6c486c59585thrg"
Jul 13 19:11:06.352639 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:11:06.352593    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-6c486c59585thrg" event={"ID":"4f31d263-8599-4d5b-8100-924cc7fdc7e9","Type":"ContainerDied","Data":"d98719d6208761166bc9c089a8be18e8e12e92cdfe3d47278ea8a97356ebcc22"}
Jul 13 19:11:06.352930 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:11:06.352651    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-6c486c59585thrg" event={"ID":"4f31d263-8599-4d5b-8100-924cc7fdc7e9","Type":"ContainerDied","Data":"6c24314aad24dd5536da842d8a5be6cbae44d4a4a8cd8fc3613b96655aedffd5"}
Jul 13 19:11:06.352930 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:11:06.352673    2565 scope.go:117] "RemoveContainer" containerID="d98719d6208761166bc9c089a8be18e8e12e92cdfe3d47278ea8a97356ebcc22"
Jul 13 19:11:06.361551 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:11:06.361536    2565 scope.go:117] "RemoveContainer" containerID="6a749e0c431741665213264eec85e8c820e24e8a3f8c641fcbea7f693aece293"
Jul 13 19:11:06.369635 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:11:06.369484    2565 scope.go:117] "RemoveContainer" containerID="d98719d6208761166bc9c089a8be18e8e12e92cdfe3d47278ea8a97356ebcc22"
Jul 13 19:11:06.369831 ip-10-0-128-215 kubenswrapper[2565]: E0713 19:11:06.369806    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d98719d6208761166bc9c089a8be18e8e12e92cdfe3d47278ea8a97356ebcc22\": container with ID starting with d98719d6208761166bc9c089a8be18e8e12e92cdfe3d47278ea8a97356ebcc22 not found: ID does not exist" containerID="d98719d6208761166bc9c089a8be18e8e12e92cdfe3d47278ea8a97356ebcc22"
Jul 13 19:11:06.369922 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:11:06.369845    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d98719d6208761166bc9c089a8be18e8e12e92cdfe3d47278ea8a97356ebcc22"} err="failed to get container status \"d98719d6208761166bc9c089a8be18e8e12e92cdfe3d47278ea8a97356ebcc22\": rpc error: code = NotFound desc = could not find container \"d98719d6208761166bc9c089a8be18e8e12e92cdfe3d47278ea8a97356ebcc22\": container with ID starting with d98719d6208761166bc9c089a8be18e8e12e92cdfe3d47278ea8a97356ebcc22 not found: ID does not exist"
Jul 13 19:11:06.369922 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:11:06.369867    2565 scope.go:117] "RemoveContainer" containerID="6a749e0c431741665213264eec85e8c820e24e8a3f8c641fcbea7f693aece293"
Jul 13 19:11:06.370155 ip-10-0-128-215 kubenswrapper[2565]: E0713 19:11:06.370131    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a749e0c431741665213264eec85e8c820e24e8a3f8c641fcbea7f693aece293\": container with ID starting with 6a749e0c431741665213264eec85e8c820e24e8a3f8c641fcbea7f693aece293 not found: ID does not exist" containerID="6a749e0c431741665213264eec85e8c820e24e8a3f8c641fcbea7f693aece293"
Jul 13 19:11:06.370255 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:11:06.370158    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a749e0c431741665213264eec85e8c820e24e8a3f8c641fcbea7f693aece293"} err="failed to get container status \"6a749e0c431741665213264eec85e8c820e24e8a3f8c641fcbea7f693aece293\": rpc error: code = NotFound desc = could not find container \"6a749e0c431741665213264eec85e8c820e24e8a3f8c641fcbea7f693aece293\": container with ID starting with 6a749e0c431741665213264eec85e8c820e24e8a3f8c641fcbea7f693aece293 not found: ID does not exist"
Jul 13 19:11:06.372065 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:11:06.372046    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-6c486c59585thrg"]
Jul 13 19:11:06.375906 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:11:06.375889    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-6c486c59585thrg"]
Jul 13 19:11:08.048309 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:11:08.048274    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4f31d263-8599-4d5b-8100-924cc7fdc7e9" path="/var/lib/kubelet/pods/4f31d263-8599-4d5b-8100-924cc7fdc7e9/volumes"
Jul 13 19:11:20.574407 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:11:20.574346    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-58b5cf756b-27d2s" podUID="5261b96e-7755-486b-89fe-7875fbfe692e" containerName="main" containerID="cri-o://ca3cfb652d9117652936e007797aa21e9266b521d220e0927c3743ac4844d2f3" gracePeriod=45
Jul 13 19:11:20.717372 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:11:20.717333    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-58b5cf756b-27d2s" podUID="5261b96e-7755-486b-89fe-7875fbfe692e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/health\": dial tcp 10.134.0.43:8000: connect: connection refused"
Jul 13 19:11:21.243919 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:11:21.243895    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-58b5cf756b-27d2s"
Jul 13 19:11:21.306569 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:11:21.306544    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5261b96e-7755-486b-89fe-7875fbfe692e-dshm\") pod \"5261b96e-7755-486b-89fe-7875fbfe692e\" (UID: \"5261b96e-7755-486b-89fe-7875fbfe692e\") "
Jul 13 19:11:21.306740 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:11:21.306590    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5261b96e-7755-486b-89fe-7875fbfe692e-kserve-provision-location\") pod \"5261b96e-7755-486b-89fe-7875fbfe692e\" (UID: \"5261b96e-7755-486b-89fe-7875fbfe692e\") "
Jul 13 19:11:21.306740 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:11:21.306618    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5261b96e-7755-486b-89fe-7875fbfe692e-model-cache\") pod \"5261b96e-7755-486b-89fe-7875fbfe692e\" (UID: \"5261b96e-7755-486b-89fe-7875fbfe692e\") "
Jul 13 19:11:21.306740 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:11:21.306679    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5261b96e-7755-486b-89fe-7875fbfe692e-tmp-dir\") pod \"5261b96e-7755-486b-89fe-7875fbfe692e\" (UID: \"5261b96e-7755-486b-89fe-7875fbfe692e\") "
Jul 13 19:11:21.306740 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:11:21.306734    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5261b96e-7755-486b-89fe-7875fbfe692e-home\") pod \"5261b96e-7755-486b-89fe-7875fbfe692e\" (UID: \"5261b96e-7755-486b-89fe-7875fbfe692e\") "
Jul 13 19:11:21.306958 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:11:21.306764    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qtzng\" (UniqueName: \"kubernetes.io/projected/5261b96e-7755-486b-89fe-7875fbfe692e-kube-api-access-qtzng\") pod \"5261b96e-7755-486b-89fe-7875fbfe692e\" (UID: \"5261b96e-7755-486b-89fe-7875fbfe692e\") "
Jul 13 19:11:21.306958 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:11:21.306796    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5261b96e-7755-486b-89fe-7875fbfe692e-tls-certs\") pod \"5261b96e-7755-486b-89fe-7875fbfe692e\" (UID: \"5261b96e-7755-486b-89fe-7875fbfe692e\") "
Jul 13 19:11:21.306958 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:11:21.306909    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5261b96e-7755-486b-89fe-7875fbfe692e-model-cache" (OuterVolumeSpecName: "model-cache") pod "5261b96e-7755-486b-89fe-7875fbfe692e" (UID: "5261b96e-7755-486b-89fe-7875fbfe692e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:11:21.307179 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:11:21.307154    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5261b96e-7755-486b-89fe-7875fbfe692e-model-cache\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 13 19:11:21.307406 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:11:21.307379    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5261b96e-7755-486b-89fe-7875fbfe692e-home" (OuterVolumeSpecName: "home") pod "5261b96e-7755-486b-89fe-7875fbfe692e" (UID: "5261b96e-7755-486b-89fe-7875fbfe692e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:11:21.309002 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:11:21.308972    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5261b96e-7755-486b-89fe-7875fbfe692e-kube-api-access-qtzng" (OuterVolumeSpecName: "kube-api-access-qtzng") pod "5261b96e-7755-486b-89fe-7875fbfe692e" (UID: "5261b96e-7755-486b-89fe-7875fbfe692e"). InnerVolumeSpecName "kube-api-access-qtzng". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 19:11:21.309355 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:11:21.309336    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5261b96e-7755-486b-89fe-7875fbfe692e-dshm" (OuterVolumeSpecName: "dshm") pod "5261b96e-7755-486b-89fe-7875fbfe692e" (UID: "5261b96e-7755-486b-89fe-7875fbfe692e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:11:21.309421 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:11:21.309363    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5261b96e-7755-486b-89fe-7875fbfe692e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5261b96e-7755-486b-89fe-7875fbfe692e" (UID: "5261b96e-7755-486b-89fe-7875fbfe692e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 19:11:21.318459 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:11:21.318424    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5261b96e-7755-486b-89fe-7875fbfe692e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5261b96e-7755-486b-89fe-7875fbfe692e" (UID: "5261b96e-7755-486b-89fe-7875fbfe692e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:11:21.377069 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:11:21.377006    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5261b96e-7755-486b-89fe-7875fbfe692e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5261b96e-7755-486b-89fe-7875fbfe692e" (UID: "5261b96e-7755-486b-89fe-7875fbfe692e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:11:21.407682 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:11:21.407659    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5261b96e-7755-486b-89fe-7875fbfe692e-kserve-provision-location\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 13 19:11:21.407682 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:11:21.407681    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5261b96e-7755-486b-89fe-7875fbfe692e-tmp-dir\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 13 19:11:21.407841 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:11:21.407691    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5261b96e-7755-486b-89fe-7875fbfe692e-home\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 13 19:11:21.407841 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:11:21.407721    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qtzng\" (UniqueName: \"kubernetes.io/projected/5261b96e-7755-486b-89fe-7875fbfe692e-kube-api-access-qtzng\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 13 19:11:21.407841 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:11:21.407731    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5261b96e-7755-486b-89fe-7875fbfe692e-tls-certs\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 13 19:11:21.407841 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:11:21.407740    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5261b96e-7755-486b-89fe-7875fbfe692e-dshm\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 13 19:11:21.430060 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:11:21.430034    2565 generic.go:356] "Generic (PLEG): container finished" podID="5261b96e-7755-486b-89fe-7875fbfe692e" containerID="ca3cfb652d9117652936e007797aa21e9266b521d220e0927c3743ac4844d2f3" exitCode=0
Jul 13 19:11:21.430149 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:11:21.430066    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-58b5cf756b-27d2s" event={"ID":"5261b96e-7755-486b-89fe-7875fbfe692e","Type":"ContainerDied","Data":"ca3cfb652d9117652936e007797aa21e9266b521d220e0927c3743ac4844d2f3"}
Jul 13 19:11:21.430149 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:11:21.430082    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-58b5cf756b-27d2s"
Jul 13 19:11:21.430149 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:11:21.430105    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-58b5cf756b-27d2s" event={"ID":"5261b96e-7755-486b-89fe-7875fbfe692e","Type":"ContainerDied","Data":"34f1ae3dc98a6855273cf144c2d25ab4d731199cf1d353fd981d76576f99fc03"}
Jul 13 19:11:21.430149 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:11:21.430126    2565 scope.go:117] "RemoveContainer" containerID="ca3cfb652d9117652936e007797aa21e9266b521d220e0927c3743ac4844d2f3"
Jul 13 19:11:21.439443 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:11:21.439428    2565 scope.go:117] "RemoveContainer" containerID="f965e6feda3bc8f6eb07a71fce3261ed485c6f27ed17b05bd8ae99db552d6dff"
Jul 13 19:11:21.447514 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:11:21.447498    2565 scope.go:117] "RemoveContainer" containerID="ca3cfb652d9117652936e007797aa21e9266b521d220e0927c3743ac4844d2f3"
Jul 13 19:11:21.447847 ip-10-0-128-215 kubenswrapper[2565]: E0713 19:11:21.447818    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca3cfb652d9117652936e007797aa21e9266b521d220e0927c3743ac4844d2f3\": container with ID starting with ca3cfb652d9117652936e007797aa21e9266b521d220e0927c3743ac4844d2f3 not found: ID does not exist" containerID="ca3cfb652d9117652936e007797aa21e9266b521d220e0927c3743ac4844d2f3"
Jul 13 19:11:21.447899 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:11:21.447859    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca3cfb652d9117652936e007797aa21e9266b521d220e0927c3743ac4844d2f3"} err="failed to get container status \"ca3cfb652d9117652936e007797aa21e9266b521d220e0927c3743ac4844d2f3\": rpc error: code = NotFound desc = could not find container \"ca3cfb652d9117652936e007797aa21e9266b521d220e0927c3743ac4844d2f3\": container with ID starting with ca3cfb652d9117652936e007797aa21e9266b521d220e0927c3743ac4844d2f3 not found: ID does not exist"
Jul 13 19:11:21.447899 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:11:21.447881    2565 scope.go:117] "RemoveContainer" containerID="f965e6feda3bc8f6eb07a71fce3261ed485c6f27ed17b05bd8ae99db552d6dff"
Jul 13 19:11:21.448133 ip-10-0-128-215 kubenswrapper[2565]: E0713 19:11:21.448115    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f965e6feda3bc8f6eb07a71fce3261ed485c6f27ed17b05bd8ae99db552d6dff\": container with ID starting with f965e6feda3bc8f6eb07a71fce3261ed485c6f27ed17b05bd8ae99db552d6dff not found: ID does not exist" containerID="f965e6feda3bc8f6eb07a71fce3261ed485c6f27ed17b05bd8ae99db552d6dff"
Jul 13 19:11:21.448189 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:11:21.448142    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f965e6feda3bc8f6eb07a71fce3261ed485c6f27ed17b05bd8ae99db552d6dff"} err="failed to get container status \"f965e6feda3bc8f6eb07a71fce3261ed485c6f27ed17b05bd8ae99db552d6dff\": rpc error: code = NotFound desc = could not find container \"f965e6feda3bc8f6eb07a71fce3261ed485c6f27ed17b05bd8ae99db552d6dff\": container with ID starting with f965e6feda3bc8f6eb07a71fce3261ed485c6f27ed17b05bd8ae99db552d6dff not found: ID does not exist"
Jul 13 19:11:21.455061 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:11:21.455042    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/auth-invalid-token-test-kserve-58b5cf756b-27d2s"]
Jul 13 19:11:21.463068 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:11:21.463046    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/auth-invalid-token-test-kserve-58b5cf756b-27d2s"]
Jul 13 19:11:22.054427 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:11:22.054395    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5261b96e-7755-486b-89fe-7875fbfe692e" path="/var/lib/kubelet/pods/5261b96e-7755-486b-89fe-7875fbfe692e/volumes"
Jul 13 19:11:41.233954 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:11:41.233919    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/auth-disabled-test-kserve-ccd54d6d4-cgwk9"]
Jul 13 19:11:41.234382 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:11:41.234282    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5261b96e-7755-486b-89fe-7875fbfe692e" containerName="storage-initializer"
Jul 13 19:11:41.234382 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:11:41.234293    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="5261b96e-7755-486b-89fe-7875fbfe692e" containerName="storage-initializer"
Jul 13 19:11:41.234382 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:11:41.234301    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f31d263-8599-4d5b-8100-924cc7fdc7e9" containerName="main"
Jul 13 19:11:41.234382 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:11:41.234306    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f31d263-8599-4d5b-8100-924cc7fdc7e9" containerName="main"
Jul 13 19:11:41.234382 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:11:41.234320    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f31d263-8599-4d5b-8100-924cc7fdc7e9" containerName="storage-initializer"
Jul 13 19:11:41.234382 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:11:41.234325    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f31d263-8599-4d5b-8100-924cc7fdc7e9" containerName="storage-initializer"
Jul 13 19:11:41.234382 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:11:41.234333    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5261b96e-7755-486b-89fe-7875fbfe692e" containerName="main"
Jul 13 19:11:41.234382 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:11:41.234338    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="5261b96e-7755-486b-89fe-7875fbfe692e" containerName="main"
Jul 13 19:11:41.234641 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:11:41.234399    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="4f31d263-8599-4d5b-8100-924cc7fdc7e9" containerName="main"
Jul 13 19:11:41.234641 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:11:41.234407    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="5261b96e-7755-486b-89fe-7875fbfe692e" containerName="main"
Jul 13 19:11:41.237450 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:11:41.237433    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-ccd54d6d4-cgwk9"
Jul 13 19:11:41.240074 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:11:41.240050    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 13 19:11:41.240563 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:11:41.240548    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 13 19:11:41.240824 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:11:41.240774    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 13 19:11:41.240824 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:11:41.240786    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-gv2sm\""
Jul 13 19:11:41.241046 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:11:41.241033    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"auth-disabled-test-kserve-self-signed-certs\""
Jul 13 19:11:41.261189 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:11:41.261170    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/92293c6c-8d0e-432f-b90b-a2659ee7ab55-home\") pod \"auth-disabled-test-kserve-ccd54d6d4-cgwk9\" (UID: \"92293c6c-8d0e-432f-b90b-a2659ee7ab55\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-ccd54d6d4-cgwk9"
Jul 13 19:11:41.261287 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:11:41.261202    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/92293c6c-8d0e-432f-b90b-a2659ee7ab55-dshm\") pod \"auth-disabled-test-kserve-ccd54d6d4-cgwk9\" (UID: \"92293c6c-8d0e-432f-b90b-a2659ee7ab55\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-ccd54d6d4-cgwk9"
Jul 13 19:11:41.261287 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:11:41.261218    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zgbs\" (UniqueName: \"kubernetes.io/projected/92293c6c-8d0e-432f-b90b-a2659ee7ab55-kube-api-access-6zgbs\") pod \"auth-disabled-test-kserve-ccd54d6d4-cgwk9\" (UID: \"92293c6c-8d0e-432f-b90b-a2659ee7ab55\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-ccd54d6d4-cgwk9"
Jul 13 19:11:41.261287 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:11:41.261237    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/92293c6c-8d0e-432f-b90b-a2659ee7ab55-model-cache\") pod \"auth-disabled-test-kserve-ccd54d6d4-cgwk9\" (UID: \"92293c6c-8d0e-432f-b90b-a2659ee7ab55\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-ccd54d6d4-cgwk9"
Jul 13 19:11:41.261419 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:11:41.261289    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/92293c6c-8d0e-432f-b90b-a2659ee7ab55-tls-certs\") pod \"auth-disabled-test-kserve-ccd54d6d4-cgwk9\" (UID: \"92293c6c-8d0e-432f-b90b-a2659ee7ab55\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-ccd54d6d4-cgwk9"
Jul 13 19:11:41.261419 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:11:41.261379    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/92293c6c-8d0e-432f-b90b-a2659ee7ab55-tmp-dir\") pod \"auth-disabled-test-kserve-ccd54d6d4-cgwk9\" (UID: \"92293c6c-8d0e-432f-b90b-a2659ee7ab55\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-ccd54d6d4-cgwk9"
Jul 13 19:11:41.261489 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:11:41.261444    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/92293c6c-8d0e-432f-b90b-a2659ee7ab55-kserve-provision-location\") pod \"auth-disabled-test-kserve-ccd54d6d4-cgwk9\" (UID: \"92293c6c-8d0e-432f-b90b-a2659ee7ab55\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-ccd54d6d4-cgwk9"
Jul 13 19:11:41.277607 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:11:41.277584    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/auth-disabled-test-kserve-ccd54d6d4-cgwk9"]
Jul 13 19:11:41.362481 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:11:41.362459    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/92293c6c-8d0e-432f-b90b-a2659ee7ab55-model-cache\") pod \"auth-disabled-test-kserve-ccd54d6d4-cgwk9\" (UID: \"92293c6c-8d0e-432f-b90b-a2659ee7ab55\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-ccd54d6d4-cgwk9"
Jul 13 19:11:41.362601 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:11:41.362488    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/92293c6c-8d0e-432f-b90b-a2659ee7ab55-tls-certs\") pod \"auth-disabled-test-kserve-ccd54d6d4-cgwk9\" (UID: \"92293c6c-8d0e-432f-b90b-a2659ee7ab55\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-ccd54d6d4-cgwk9"
Jul 13 19:11:41.362601 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:11:41.362522    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/92293c6c-8d0e-432f-b90b-a2659ee7ab55-tmp-dir\") pod \"auth-disabled-test-kserve-ccd54d6d4-cgwk9\" (UID: \"92293c6c-8d0e-432f-b90b-a2659ee7ab55\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-ccd54d6d4-cgwk9"
Jul 13 19:11:41.362696 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:11:41.362642    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/92293c6c-8d0e-432f-b90b-a2659ee7ab55-kserve-provision-location\") pod \"auth-disabled-test-kserve-ccd54d6d4-cgwk9\" (UID: \"92293c6c-8d0e-432f-b90b-a2659ee7ab55\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-ccd54d6d4-cgwk9"
Jul 13 19:11:41.362796 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:11:41.362777    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/92293c6c-8d0e-432f-b90b-a2659ee7ab55-home\") pod \"auth-disabled-test-kserve-ccd54d6d4-cgwk9\" (UID: \"92293c6c-8d0e-432f-b90b-a2659ee7ab55\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-ccd54d6d4-cgwk9"
Jul 13 19:11:41.362842 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:11:41.362823    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/92293c6c-8d0e-432f-b90b-a2659ee7ab55-dshm\") pod \"auth-disabled-test-kserve-ccd54d6d4-cgwk9\" (UID: \"92293c6c-8d0e-432f-b90b-a2659ee7ab55\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-ccd54d6d4-cgwk9"
Jul 13 19:11:41.362899 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:11:41.362848    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zgbs\" (UniqueName: \"kubernetes.io/projected/92293c6c-8d0e-432f-b90b-a2659ee7ab55-kube-api-access-6zgbs\") pod \"auth-disabled-test-kserve-ccd54d6d4-cgwk9\" (UID: \"92293c6c-8d0e-432f-b90b-a2659ee7ab55\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-ccd54d6d4-cgwk9"
Jul 13 19:11:41.362899 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:11:41.362850    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/92293c6c-8d0e-432f-b90b-a2659ee7ab55-model-cache\") pod \"auth-disabled-test-kserve-ccd54d6d4-cgwk9\" (UID: \"92293c6c-8d0e-432f-b90b-a2659ee7ab55\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-ccd54d6d4-cgwk9"
Jul 13 19:11:41.362899 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:11:41.362868    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/92293c6c-8d0e-432f-b90b-a2659ee7ab55-tmp-dir\") pod \"auth-disabled-test-kserve-ccd54d6d4-cgwk9\" (UID: \"92293c6c-8d0e-432f-b90b-a2659ee7ab55\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-ccd54d6d4-cgwk9"
Jul 13 19:11:41.363053 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:11:41.362923    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/92293c6c-8d0e-432f-b90b-a2659ee7ab55-kserve-provision-location\") pod \"auth-disabled-test-kserve-ccd54d6d4-cgwk9\" (UID: \"92293c6c-8d0e-432f-b90b-a2659ee7ab55\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-ccd54d6d4-cgwk9"
Jul 13 19:11:41.363121 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:11:41.363102    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/92293c6c-8d0e-432f-b90b-a2659ee7ab55-home\") pod \"auth-disabled-test-kserve-ccd54d6d4-cgwk9\" (UID: \"92293c6c-8d0e-432f-b90b-a2659ee7ab55\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-ccd54d6d4-cgwk9"
Jul 13 19:11:41.365067 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:11:41.365045    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/92293c6c-8d0e-432f-b90b-a2659ee7ab55-dshm\") pod \"auth-disabled-test-kserve-ccd54d6d4-cgwk9\" (UID: \"92293c6c-8d0e-432f-b90b-a2659ee7ab55\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-ccd54d6d4-cgwk9"
Jul 13 19:11:41.365141 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:11:41.365095    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/92293c6c-8d0e-432f-b90b-a2659ee7ab55-tls-certs\") pod \"auth-disabled-test-kserve-ccd54d6d4-cgwk9\" (UID: \"92293c6c-8d0e-432f-b90b-a2659ee7ab55\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-ccd54d6d4-cgwk9"
Jul 13 19:11:41.377303 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:11:41.377281    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zgbs\" (UniqueName: \"kubernetes.io/projected/92293c6c-8d0e-432f-b90b-a2659ee7ab55-kube-api-access-6zgbs\") pod \"auth-disabled-test-kserve-ccd54d6d4-cgwk9\" (UID: \"92293c6c-8d0e-432f-b90b-a2659ee7ab55\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-ccd54d6d4-cgwk9"
Jul 13 19:11:41.548002 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:11:41.547972    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-ccd54d6d4-cgwk9"
Jul 13 19:11:41.701485 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:11:41.701434    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/auth-disabled-test-kserve-ccd54d6d4-cgwk9"]
Jul 13 19:11:41.703817 ip-10-0-128-215 kubenswrapper[2565]: W0713 19:11:41.703783    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92293c6c_8d0e_432f_b90b_a2659ee7ab55.slice/crio-a041b0d580135e2baf251fe0b5d7a86504d122b4231a20a4c20811cc0342aa0e WatchSource:0}: Error finding container a041b0d580135e2baf251fe0b5d7a86504d122b4231a20a4c20811cc0342aa0e: Status 404 returned error can't find the container with id a041b0d580135e2baf251fe0b5d7a86504d122b4231a20a4c20811cc0342aa0e
Jul 13 19:11:42.530922 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:11:42.530887    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-ccd54d6d4-cgwk9" event={"ID":"92293c6c-8d0e-432f-b90b-a2659ee7ab55","Type":"ContainerStarted","Data":"1f0c41664078b53ae7771c4d2f2eb3bbaa7cbc0cac6e329b607f9ff7d4d43225"}
Jul 13 19:11:42.531303 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:11:42.530964    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-ccd54d6d4-cgwk9" event={"ID":"92293c6c-8d0e-432f-b90b-a2659ee7ab55","Type":"ContainerStarted","Data":"a041b0d580135e2baf251fe0b5d7a86504d122b4231a20a4c20811cc0342aa0e"}
Jul 13 19:11:45.550713 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:11:45.550673    2565 generic.go:356] "Generic (PLEG): container finished" podID="92293c6c-8d0e-432f-b90b-a2659ee7ab55" containerID="1f0c41664078b53ae7771c4d2f2eb3bbaa7cbc0cac6e329b607f9ff7d4d43225" exitCode=0
Jul 13 19:11:45.551098 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:11:45.550747    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-ccd54d6d4-cgwk9" event={"ID":"92293c6c-8d0e-432f-b90b-a2659ee7ab55","Type":"ContainerDied","Data":"1f0c41664078b53ae7771c4d2f2eb3bbaa7cbc0cac6e329b607f9ff7d4d43225"}
Jul 13 19:11:46.560614 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:11:46.560571    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-ccd54d6d4-cgwk9" event={"ID":"92293c6c-8d0e-432f-b90b-a2659ee7ab55","Type":"ContainerStarted","Data":"f3394c970fa757d42c5689a51de1a7f4cbc470caba3d6775de91d3f4e9fd9fdd"}
Jul 13 19:11:47.548590 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:11:47.548555    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-ccd54d6d4-cgwk9"
Jul 13 19:11:51.548885 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:11:51.548853    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-ccd54d6d4-cgwk9"
Jul 13 19:11:51.550600 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:11:51.550571    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-ccd54d6d4-cgwk9" podUID="92293c6c-8d0e-432f-b90b-a2659ee7ab55" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": dial tcp 10.134.0.45:8000: connect: connection refused"
Jul 13 19:11:55.421684 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:11:55.421633    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-ccd54d6d4-cgwk9" podStartSLOduration=11.403470456 podStartE2EDuration="14.421616964s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.018146508s" isStatefulPod=false podCreationTimestamp="2026-07-13 19:11:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 19:11:46.605988944 +0000 UTC m=+1293.131122396" watchObservedRunningTime="2026-07-13 19:11:55.421616964 +0000 UTC m=+1301.946750379"
Jul 13 19:11:55.422088 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:11:55.421959    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schecjg9g"]
Jul 13 19:11:55.440131 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:11:55.440106    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schecjg9g"
Jul 13 19:11:55.443907 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:11:55.443593    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schecjg9g"]
Jul 13 19:11:55.443907 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:11:55.443863    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv08544b88a8d963ffd553cc1f3ed82d16-kserve-self-signed-certs\""
Jul 13 19:11:55.446757 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:11:55.446721    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-epp-sa-dockercfg-pprtb\""
Jul 13 19:11:55.602369 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:11:55.602337    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/3c00868f-4b02-43eb-aff6-e42cf604eec8-tokenizer-uds\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schecjg9g\" (UID: \"3c00868f-4b02-43eb-aff6-e42cf604eec8\") " pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schecjg9g"
Jul 13 19:11:55.602545 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:11:55.602383    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkbtq\" (UniqueName: \"kubernetes.io/projected/3c00868f-4b02-43eb-aff6-e42cf604eec8-kube-api-access-xkbtq\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schecjg9g\" (UID: \"3c00868f-4b02-43eb-aff6-e42cf604eec8\") " pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schecjg9g"
Jul 13 19:11:55.602545 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:11:55.602419    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c00868f-4b02-43eb-aff6-e42cf604eec8-tls-certs\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schecjg9g\" (UID: \"3c00868f-4b02-43eb-aff6-e42cf604eec8\") " pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schecjg9g"
Jul 13 19:11:55.703170 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:11:55.703073    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkbtq\" (UniqueName: \"kubernetes.io/projected/3c00868f-4b02-43eb-aff6-e42cf604eec8-kube-api-access-xkbtq\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schecjg9g\" (UID: \"3c00868f-4b02-43eb-aff6-e42cf604eec8\") " pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schecjg9g"
Jul 13 19:11:55.703170 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:11:55.703138    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c00868f-4b02-43eb-aff6-e42cf604eec8-tls-certs\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schecjg9g\" (UID: \"3c00868f-4b02-43eb-aff6-e42cf604eec8\") " pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schecjg9g"
Jul 13 19:11:55.703443 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:11:55.703231    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/3c00868f-4b02-43eb-aff6-e42cf604eec8-tokenizer-uds\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schecjg9g\" (UID: \"3c00868f-4b02-43eb-aff6-e42cf604eec8\") " pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schecjg9g"
Jul 13 19:11:55.703631 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:11:55.703606    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/3c00868f-4b02-43eb-aff6-e42cf604eec8-tokenizer-uds\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schecjg9g\" (UID: \"3c00868f-4b02-43eb-aff6-e42cf604eec8\") " pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schecjg9g"
Jul 13 19:11:55.706083 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:11:55.706052    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c00868f-4b02-43eb-aff6-e42cf604eec8-tls-certs\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schecjg9g\" (UID: \"3c00868f-4b02-43eb-aff6-e42cf604eec8\") " pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schecjg9g"
Jul 13 19:11:55.712101 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:11:55.712078    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkbtq\" (UniqueName: \"kubernetes.io/projected/3c00868f-4b02-43eb-aff6-e42cf604eec8-kube-api-access-xkbtq\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schecjg9g\" (UID: \"3c00868f-4b02-43eb-aff6-e42cf604eec8\") " pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schecjg9g"
Jul 13 19:11:55.756984 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:11:55.756948    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schecjg9g"
Jul 13 19:11:55.918934 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:11:55.918910    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schecjg9g"]
Jul 13 19:11:55.921936 ip-10-0-128-215 kubenswrapper[2565]: W0713 19:11:55.921908    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c00868f_4b02_43eb_aff6_e42cf604eec8.slice/crio-b7394a113c1d11f920c948d04829ea63b9538151ca15ca0c36e2226a8bd8326f WatchSource:0}: Error finding container b7394a113c1d11f920c948d04829ea63b9538151ca15ca0c36e2226a8bd8326f: Status 404 returned error can't find the container with id b7394a113c1d11f920c948d04829ea63b9538151ca15ca0c36e2226a8bd8326f
Jul 13 19:11:56.615514 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:11:56.615481    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schecjg9g" event={"ID":"3c00868f-4b02-43eb-aff6-e42cf604eec8","Type":"ContainerStarted","Data":"59ace6c7ab473411e3b6840969a4ef801aa0ac61bb0b40e7b67036221dfb64ec"}
Jul 13 19:11:56.615996 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:11:56.615536    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schecjg9g" event={"ID":"3c00868f-4b02-43eb-aff6-e42cf604eec8","Type":"ContainerStarted","Data":"b7394a113c1d11f920c948d04829ea63b9538151ca15ca0c36e2226a8bd8326f"}
Jul 13 19:11:56.615996 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:11:56.615584    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schecjg9g"
Jul 13 19:11:56.636626 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:11:56.636556    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schecjg9g" podStartSLOduration=1.636536554 podStartE2EDuration="1.636536554s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 19:11:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 19:11:56.635579346 +0000 UTC m=+1303.160712782" watchObservedRunningTime="2026-07-13 19:11:56.636536554 +0000 UTC m=+1303.161669974"
Jul 13 19:12:01.549103 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:12:01.549062    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-ccd54d6d4-cgwk9" podUID="92293c6c-8d0e-432f-b90b-a2659ee7ab55" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": dial tcp 10.134.0.45:8000: connect: connection refused"
Jul 13 19:12:11.548466 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:12:11.548420    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-ccd54d6d4-cgwk9" podUID="92293c6c-8d0e-432f-b90b-a2659ee7ab55" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": dial tcp 10.134.0.45:8000: connect: connection refused"
Jul 13 19:12:14.038920 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:12:14.038893    2565 scope.go:117] "RemoveContainer" containerID="ba1f6d32e456075beb1e675e4b0af155fd2aae6bfa8b2dd53fb8234f1479350f"
Jul 13 19:12:21.548783 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:12:21.548735    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-ccd54d6d4-cgwk9" podUID="92293c6c-8d0e-432f-b90b-a2659ee7ab55" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": dial tcp 10.134.0.45:8000: connect: connection refused"
Jul 13 19:12:27.622836 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:12:27.622797    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schecjg9g"
Jul 13 19:12:31.548960 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:12:31.548908    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-ccd54d6d4-cgwk9" podUID="92293c6c-8d0e-432f-b90b-a2659ee7ab55" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": dial tcp 10.134.0.45:8000: connect: connection refused"
Jul 13 19:12:33.004830 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:12:33.004798    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schecjg9g"]
Jul 13 19:12:33.014068 ip-10-0-128-215 kubenswrapper[2565]: E0713 19:12:33.014031    2565 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 13 19:12:33.014068 ip-10-0-128-215 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 13 19:12:33.014068 ip-10-0-128-215 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Jul 13 19:12:33.014068 ip-10-0-128-215 kubenswrapper[2565]:  > containerID="59ace6c7ab473411e3b6840969a4ef801aa0ac61bb0b40e7b67036221dfb64ec" cmd=["/bin/sleep","15"]
Jul 13 19:12:33.014369 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:12:33.014083    2565 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schecjg9g" message="" err=<
Jul 13 19:12:33.014369 ip-10-0-128-215 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 13 19:12:33.014369 ip-10-0-128-215 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Jul 13 19:12:33.014369 ip-10-0-128-215 kubenswrapper[2565]:  >
Jul 13 19:12:33.014369 ip-10-0-128-215 kubenswrapper[2565]: E0713 19:12:33.014096    2565 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 13 19:12:33.014369 ip-10-0-128-215 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 13 19:12:33.014369 ip-10-0-128-215 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Jul 13 19:12:33.014369 ip-10-0-128-215 kubenswrapper[2565]:  > pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schecjg9g" podUID="3c00868f-4b02-43eb-aff6-e42cf604eec8" containerName="main" containerID="cri-o://59ace6c7ab473411e3b6840969a4ef801aa0ac61bb0b40e7b67036221dfb64ec"
Jul 13 19:12:33.014369 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:12:33.014135    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schecjg9g" podUID="3c00868f-4b02-43eb-aff6-e42cf604eec8" containerName="main" containerID="cri-o://59ace6c7ab473411e3b6840969a4ef801aa0ac61bb0b40e7b67036221dfb64ec" gracePeriod=60
Jul 13 19:12:33.263611 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:12:33.263585    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schecjg9g"
Jul 13 19:12:33.363532 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:12:33.363493    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/3c00868f-4b02-43eb-aff6-e42cf604eec8-tokenizer-uds\") pod \"3c00868f-4b02-43eb-aff6-e42cf604eec8\" (UID: \"3c00868f-4b02-43eb-aff6-e42cf604eec8\") "
Jul 13 19:12:33.363760 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:12:33.363542    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xkbtq\" (UniqueName: \"kubernetes.io/projected/3c00868f-4b02-43eb-aff6-e42cf604eec8-kube-api-access-xkbtq\") pod \"3c00868f-4b02-43eb-aff6-e42cf604eec8\" (UID: \"3c00868f-4b02-43eb-aff6-e42cf604eec8\") "
Jul 13 19:12:33.363760 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:12:33.363593    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c00868f-4b02-43eb-aff6-e42cf604eec8-tls-certs\") pod \"3c00868f-4b02-43eb-aff6-e42cf604eec8\" (UID: \"3c00868f-4b02-43eb-aff6-e42cf604eec8\") "
Jul 13 19:12:33.363886 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:12:33.363783    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c00868f-4b02-43eb-aff6-e42cf604eec8-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "3c00868f-4b02-43eb-aff6-e42cf604eec8" (UID: "3c00868f-4b02-43eb-aff6-e42cf604eec8"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:12:33.363939 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:12:33.363919    2565 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/3c00868f-4b02-43eb-aff6-e42cf604eec8-tokenizer-uds\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 13 19:12:33.365819 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:12:33.365792    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c00868f-4b02-43eb-aff6-e42cf604eec8-kube-api-access-xkbtq" (OuterVolumeSpecName: "kube-api-access-xkbtq") pod "3c00868f-4b02-43eb-aff6-e42cf604eec8" (UID: "3c00868f-4b02-43eb-aff6-e42cf604eec8"). InnerVolumeSpecName "kube-api-access-xkbtq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 19:12:33.365922 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:12:33.365902    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c00868f-4b02-43eb-aff6-e42cf604eec8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3c00868f-4b02-43eb-aff6-e42cf604eec8" (UID: "3c00868f-4b02-43eb-aff6-e42cf604eec8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 19:12:33.465292 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:12:33.465232    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c00868f-4b02-43eb-aff6-e42cf604eec8-tls-certs\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 13 19:12:33.465292 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:12:33.465288    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xkbtq\" (UniqueName: \"kubernetes.io/projected/3c00868f-4b02-43eb-aff6-e42cf604eec8-kube-api-access-xkbtq\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 13 19:12:33.816910 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:12:33.816874    2565 generic.go:356] "Generic (PLEG): container finished" podID="3c00868f-4b02-43eb-aff6-e42cf604eec8" containerID="59ace6c7ab473411e3b6840969a4ef801aa0ac61bb0b40e7b67036221dfb64ec" exitCode=0
Jul 13 19:12:33.817090 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:12:33.816921    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schecjg9g"
Jul 13 19:12:33.817090 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:12:33.816961    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schecjg9g" event={"ID":"3c00868f-4b02-43eb-aff6-e42cf604eec8","Type":"ContainerDied","Data":"59ace6c7ab473411e3b6840969a4ef801aa0ac61bb0b40e7b67036221dfb64ec"}
Jul 13 19:12:33.817090 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:12:33.817012    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schecjg9g" event={"ID":"3c00868f-4b02-43eb-aff6-e42cf604eec8","Type":"ContainerDied","Data":"b7394a113c1d11f920c948d04829ea63b9538151ca15ca0c36e2226a8bd8326f"}
Jul 13 19:12:33.817090 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:12:33.817035    2565 scope.go:117] "RemoveContainer" containerID="59ace6c7ab473411e3b6840969a4ef801aa0ac61bb0b40e7b67036221dfb64ec"
Jul 13 19:12:33.827077 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:12:33.827055    2565 scope.go:117] "RemoveContainer" containerID="59ace6c7ab473411e3b6840969a4ef801aa0ac61bb0b40e7b67036221dfb64ec"
Jul 13 19:12:33.827337 ip-10-0-128-215 kubenswrapper[2565]: E0713 19:12:33.827317    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59ace6c7ab473411e3b6840969a4ef801aa0ac61bb0b40e7b67036221dfb64ec\": container with ID starting with 59ace6c7ab473411e3b6840969a4ef801aa0ac61bb0b40e7b67036221dfb64ec not found: ID does not exist" containerID="59ace6c7ab473411e3b6840969a4ef801aa0ac61bb0b40e7b67036221dfb64ec"
Jul 13 19:12:33.827436 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:12:33.827343    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59ace6c7ab473411e3b6840969a4ef801aa0ac61bb0b40e7b67036221dfb64ec"} err="failed to get container status \"59ace6c7ab473411e3b6840969a4ef801aa0ac61bb0b40e7b67036221dfb64ec\": rpc error: code = NotFound desc = could not find container \"59ace6c7ab473411e3b6840969a4ef801aa0ac61bb0b40e7b67036221dfb64ec\": container with ID starting with 59ace6c7ab473411e3b6840969a4ef801aa0ac61bb0b40e7b67036221dfb64ec not found: ID does not exist"
Jul 13 19:12:33.837633 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:12:33.837612    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schecjg9g"]
Jul 13 19:12:33.841221 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:12:33.841203    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schecjg9g"]
Jul 13 19:12:34.049482 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:12:34.049451    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3c00868f-4b02-43eb-aff6-e42cf604eec8" path="/var/lib/kubelet/pods/3c00868f-4b02-43eb-aff6-e42cf604eec8/volumes"
Jul 13 19:12:41.549088 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:12:41.549039    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-ccd54d6d4-cgwk9" podUID="92293c6c-8d0e-432f-b90b-a2659ee7ab55" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": dial tcp 10.134.0.45:8000: connect: connection refused"
Jul 13 19:12:51.549164 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:12:51.549117    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-ccd54d6d4-cgwk9" podUID="92293c6c-8d0e-432f-b90b-a2659ee7ab55" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": dial tcp 10.134.0.45:8000: connect: connection refused"
Jul 13 19:13:01.548667 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:13:01.548628    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-ccd54d6d4-cgwk9" podUID="92293c6c-8d0e-432f-b90b-a2659ee7ab55" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": dial tcp 10.134.0.45:8000: connect: connection refused"
Jul 13 19:13:11.549385 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:13:11.549336    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-ccd54d6d4-cgwk9" podUID="92293c6c-8d0e-432f-b90b-a2659ee7ab55" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": dial tcp 10.134.0.45:8000: connect: connection refused"
Jul 13 19:13:18.999364 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:13:18.999321    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79c7c55747-lfmdf"]
Jul 13 19:13:18.999925 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:13:18.999905    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c00868f-4b02-43eb-aff6-e42cf604eec8" containerName="main"
Jul 13 19:13:19.000022 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:13:18.999930    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c00868f-4b02-43eb-aff6-e42cf604eec8" containerName="main"
Jul 13 19:13:19.000080 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:13:19.000046    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="3c00868f-4b02-43eb-aff6-e42cf604eec8" containerName="main"
Jul 13 19:13:19.003060 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:13:19.003038    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79c7c55747-lfmdf"
Jul 13 19:13:19.006146 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:13:19.006121    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 13 19:13:19.006251 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:13:19.006121    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv3e414c2ba058a022dfd694dbcbac5b51-kserve-self-signed-certs\""
Jul 13 19:13:19.020376 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:13:19.020350    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79c7c55747-lfmdf"]
Jul 13 19:13:19.090001 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:13:19.089968    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwxgn\" (UniqueName: \"kubernetes.io/projected/b52ab7ba-8876-4a5a-afe8-c4ed84200beb-kube-api-access-zwxgn\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79c7c55747-lfmdf\" (UID: \"b52ab7ba-8876-4a5a-afe8-c4ed84200beb\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79c7c55747-lfmdf"
Jul 13 19:13:19.090001 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:13:19.090008    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b52ab7ba-8876-4a5a-afe8-c4ed84200beb-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79c7c55747-lfmdf\" (UID: \"b52ab7ba-8876-4a5a-afe8-c4ed84200beb\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79c7c55747-lfmdf"
Jul 13 19:13:19.090245 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:13:19.090046    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b52ab7ba-8876-4a5a-afe8-c4ed84200beb-home\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79c7c55747-lfmdf\" (UID: \"b52ab7ba-8876-4a5a-afe8-c4ed84200beb\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79c7c55747-lfmdf"
Jul 13 19:13:19.090245 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:13:19.090063    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b52ab7ba-8876-4a5a-afe8-c4ed84200beb-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79c7c55747-lfmdf\" (UID: \"b52ab7ba-8876-4a5a-afe8-c4ed84200beb\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79c7c55747-lfmdf"
Jul 13 19:13:19.090245 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:13:19.090080    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b52ab7ba-8876-4a5a-afe8-c4ed84200beb-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79c7c55747-lfmdf\" (UID: \"b52ab7ba-8876-4a5a-afe8-c4ed84200beb\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79c7c55747-lfmdf"
Jul 13 19:13:19.090245 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:13:19.090105    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b52ab7ba-8876-4a5a-afe8-c4ed84200beb-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79c7c55747-lfmdf\" (UID: \"b52ab7ba-8876-4a5a-afe8-c4ed84200beb\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79c7c55747-lfmdf"
Jul 13 19:13:19.090245 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:13:19.090197    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b52ab7ba-8876-4a5a-afe8-c4ed84200beb-dshm\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79c7c55747-lfmdf\" (UID: \"b52ab7ba-8876-4a5a-afe8-c4ed84200beb\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79c7c55747-lfmdf"
Jul 13 19:13:19.090245 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:13:19.090224    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b52ab7ba-8876-4a5a-afe8-c4ed84200beb-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79c7c55747-lfmdf\" (UID: \"b52ab7ba-8876-4a5a-afe8-c4ed84200beb\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79c7c55747-lfmdf"
Jul 13 19:13:19.191615 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:13:19.191569    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b52ab7ba-8876-4a5a-afe8-c4ed84200beb-home\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79c7c55747-lfmdf\" (UID: \"b52ab7ba-8876-4a5a-afe8-c4ed84200beb\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79c7c55747-lfmdf"
Jul 13 19:13:19.191615 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:13:19.191617    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b52ab7ba-8876-4a5a-afe8-c4ed84200beb-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79c7c55747-lfmdf\" (UID: \"b52ab7ba-8876-4a5a-afe8-c4ed84200beb\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79c7c55747-lfmdf"
Jul 13 19:13:19.191917 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:13:19.191644    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b52ab7ba-8876-4a5a-afe8-c4ed84200beb-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79c7c55747-lfmdf\" (UID: \"b52ab7ba-8876-4a5a-afe8-c4ed84200beb\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79c7c55747-lfmdf"
Jul 13 19:13:19.191917 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:13:19.191666    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b52ab7ba-8876-4a5a-afe8-c4ed84200beb-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79c7c55747-lfmdf\" (UID: \"b52ab7ba-8876-4a5a-afe8-c4ed84200beb\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79c7c55747-lfmdf"
Jul 13 19:13:19.191917 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:13:19.191759    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b52ab7ba-8876-4a5a-afe8-c4ed84200beb-dshm\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79c7c55747-lfmdf\" (UID: \"b52ab7ba-8876-4a5a-afe8-c4ed84200beb\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79c7c55747-lfmdf"
Jul 13 19:13:19.191917 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:13:19.191868    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b52ab7ba-8876-4a5a-afe8-c4ed84200beb-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79c7c55747-lfmdf\" (UID: \"b52ab7ba-8876-4a5a-afe8-c4ed84200beb\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79c7c55747-lfmdf"
Jul 13 19:13:19.192120 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:13:19.192017    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwxgn\" (UniqueName: \"kubernetes.io/projected/b52ab7ba-8876-4a5a-afe8-c4ed84200beb-kube-api-access-zwxgn\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79c7c55747-lfmdf\" (UID: \"b52ab7ba-8876-4a5a-afe8-c4ed84200beb\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79c7c55747-lfmdf"
Jul 13 19:13:19.192120 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:13:19.192073    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b52ab7ba-8876-4a5a-afe8-c4ed84200beb-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79c7c55747-lfmdf\" (UID: \"b52ab7ba-8876-4a5a-afe8-c4ed84200beb\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79c7c55747-lfmdf"
Jul 13 19:13:19.192120 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:13:19.192090    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b52ab7ba-8876-4a5a-afe8-c4ed84200beb-home\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79c7c55747-lfmdf\" (UID: \"b52ab7ba-8876-4a5a-afe8-c4ed84200beb\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79c7c55747-lfmdf"
Jul 13 19:13:19.192120 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:13:19.192107    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b52ab7ba-8876-4a5a-afe8-c4ed84200beb-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79c7c55747-lfmdf\" (UID: \"b52ab7ba-8876-4a5a-afe8-c4ed84200beb\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79c7c55747-lfmdf"
Jul 13 19:13:19.192323 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:13:19.192162    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b52ab7ba-8876-4a5a-afe8-c4ed84200beb-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79c7c55747-lfmdf\" (UID: \"b52ab7ba-8876-4a5a-afe8-c4ed84200beb\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79c7c55747-lfmdf"
Jul 13 19:13:19.192494 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:13:19.192464    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b52ab7ba-8876-4a5a-afe8-c4ed84200beb-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79c7c55747-lfmdf\" (UID: \"b52ab7ba-8876-4a5a-afe8-c4ed84200beb\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79c7c55747-lfmdf"
Jul 13 19:13:19.192625 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:13:19.192575    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b52ab7ba-8876-4a5a-afe8-c4ed84200beb-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79c7c55747-lfmdf\" (UID: \"b52ab7ba-8876-4a5a-afe8-c4ed84200beb\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79c7c55747-lfmdf"
Jul 13 19:13:19.194599 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:13:19.194572    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b52ab7ba-8876-4a5a-afe8-c4ed84200beb-dshm\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79c7c55747-lfmdf\" (UID: \"b52ab7ba-8876-4a5a-afe8-c4ed84200beb\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79c7c55747-lfmdf"
Jul 13 19:13:19.194853 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:13:19.194835    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b52ab7ba-8876-4a5a-afe8-c4ed84200beb-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79c7c55747-lfmdf\" (UID: \"b52ab7ba-8876-4a5a-afe8-c4ed84200beb\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79c7c55747-lfmdf"
Jul 13 19:13:19.203743 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:13:19.203720    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwxgn\" (UniqueName: \"kubernetes.io/projected/b52ab7ba-8876-4a5a-afe8-c4ed84200beb-kube-api-access-zwxgn\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79c7c55747-lfmdf\" (UID: \"b52ab7ba-8876-4a5a-afe8-c4ed84200beb\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79c7c55747-lfmdf"
Jul 13 19:13:19.314108 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:13:19.314068    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79c7c55747-lfmdf"
Jul 13 19:13:19.465269 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:13:19.465240    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79c7c55747-lfmdf"]
Jul 13 19:13:19.467087 ip-10-0-128-215 kubenswrapper[2565]: W0713 19:13:19.467057    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb52ab7ba_8876_4a5a_afe8_c4ed84200beb.slice/crio-d76913aa056ed6a264c62627256563fd13c453af5ac72831093defcc07de46b5 WatchSource:0}: Error finding container d76913aa056ed6a264c62627256563fd13c453af5ac72831093defcc07de46b5: Status 404 returned error can't find the container with id d76913aa056ed6a264c62627256563fd13c453af5ac72831093defcc07de46b5
Jul 13 19:13:20.070666 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:13:20.070630    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79c7c55747-lfmdf" event={"ID":"b52ab7ba-8876-4a5a-afe8-c4ed84200beb","Type":"ContainerStarted","Data":"5f94646180cfd466d27c560dc17166dffb03a8819f21ae5802c79750cd1fa7dc"}
Jul 13 19:13:20.071083 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:13:20.070688    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79c7c55747-lfmdf" event={"ID":"b52ab7ba-8876-4a5a-afe8-c4ed84200beb","Type":"ContainerStarted","Data":"d76913aa056ed6a264c62627256563fd13c453af5ac72831093defcc07de46b5"}
Jul 13 19:13:21.548901 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:13:21.548841    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-ccd54d6d4-cgwk9" podUID="92293c6c-8d0e-432f-b90b-a2659ee7ab55" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": dial tcp 10.134.0.45:8000: connect: connection refused"
Jul 13 19:13:24.101826 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:13:24.101766    2565 generic.go:356] "Generic (PLEG): container finished" podID="b52ab7ba-8876-4a5a-afe8-c4ed84200beb" containerID="5f94646180cfd466d27c560dc17166dffb03a8819f21ae5802c79750cd1fa7dc" exitCode=0
Jul 13 19:13:24.102333 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:13:24.101833    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79c7c55747-lfmdf" event={"ID":"b52ab7ba-8876-4a5a-afe8-c4ed84200beb","Type":"ContainerDied","Data":"5f94646180cfd466d27c560dc17166dffb03a8819f21ae5802c79750cd1fa7dc"}
Jul 13 19:13:25.110213 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:13:25.110163    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79c7c55747-lfmdf" event={"ID":"b52ab7ba-8876-4a5a-afe8-c4ed84200beb","Type":"ContainerStarted","Data":"16e1decef4d0e439378e5a2de4be9d838215e6f40b3760a06839ecf7177d4f21"}
Jul 13 19:13:25.139109 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:13:25.139038    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79c7c55747-lfmdf" podStartSLOduration=3.5848624730000003 podStartE2EDuration="7.13901564s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.554153167s" isStatefulPod=false podCreationTimestamp="2026-07-13 19:13:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 19:13:25.138136467 +0000 UTC m=+1391.663269909" watchObservedRunningTime="2026-07-13 19:13:25.13901564 +0000 UTC m=+1391.664149057"
Jul 13 19:13:25.314748 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:13:25.314692    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79c7c55747-lfmdf"
Jul 13 19:13:29.314876 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:13:29.314835    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79c7c55747-lfmdf"
Jul 13 19:13:29.316815 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:13:29.316784    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79c7c55747-lfmdf" podUID="b52ab7ba-8876-4a5a-afe8-c4ed84200beb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": dial tcp 10.134.0.47:8000: connect: connection refused"
Jul 13 19:13:31.548655 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:13:31.548607    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-ccd54d6d4-cgwk9" podUID="92293c6c-8d0e-432f-b90b-a2659ee7ab55" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": dial tcp 10.134.0.45:8000: connect: connection refused"
Jul 13 19:13:39.314914 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:13:39.314867    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79c7c55747-lfmdf" podUID="b52ab7ba-8876-4a5a-afe8-c4ed84200beb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": dial tcp 10.134.0.47:8000: connect: connection refused"
Jul 13 19:13:41.565605 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:13:41.565567    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-ccd54d6d4-cgwk9"
Jul 13 19:13:41.579191 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:13:41.579157    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-ccd54d6d4-cgwk9"
Jul 13 19:13:47.395175 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:13:47.395140    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/auth-disabled-test-kserve-ccd54d6d4-cgwk9"]
Jul 13 19:13:49.314659 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:13:49.314612    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79c7c55747-lfmdf" podUID="b52ab7ba-8876-4a5a-afe8-c4ed84200beb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": dial tcp 10.134.0.47:8000: connect: connection refused"
Jul 13 19:13:59.314993 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:13:59.314924    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79c7c55747-lfmdf" podUID="b52ab7ba-8876-4a5a-afe8-c4ed84200beb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": dial tcp 10.134.0.47:8000: connect: connection refused"
Jul 13 19:14:02.422507 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:14:02.422411    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-ccd54d6d4-cgwk9" podUID="92293c6c-8d0e-432f-b90b-a2659ee7ab55" containerName="main" containerID="cri-o://f3394c970fa757d42c5689a51de1a7f4cbc470caba3d6775de91d3f4e9fd9fdd" gracePeriod=45
Jul 13 19:14:02.566796 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:14:02.566735    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-ccd54d6d4-cgwk9" podUID="92293c6c-8d0e-432f-b90b-a2659ee7ab55" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": dial tcp 10.134.0.45:8000: connect: connection refused"
Jul 13 19:14:03.194622 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:14:03.194588    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-ccd54d6d4-cgwk9"
Jul 13 19:14:03.335151 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:14:03.335123    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/92293c6c-8d0e-432f-b90b-a2659ee7ab55-model-cache\") pod \"92293c6c-8d0e-432f-b90b-a2659ee7ab55\" (UID: \"92293c6c-8d0e-432f-b90b-a2659ee7ab55\") "
Jul 13 19:14:03.335361 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:14:03.335162    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/92293c6c-8d0e-432f-b90b-a2659ee7ab55-dshm\") pod \"92293c6c-8d0e-432f-b90b-a2659ee7ab55\" (UID: \"92293c6c-8d0e-432f-b90b-a2659ee7ab55\") "
Jul 13 19:14:03.335361 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:14:03.335180    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/92293c6c-8d0e-432f-b90b-a2659ee7ab55-tls-certs\") pod \"92293c6c-8d0e-432f-b90b-a2659ee7ab55\" (UID: \"92293c6c-8d0e-432f-b90b-a2659ee7ab55\") "
Jul 13 19:14:03.335361 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:14:03.335204    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/92293c6c-8d0e-432f-b90b-a2659ee7ab55-tmp-dir\") pod \"92293c6c-8d0e-432f-b90b-a2659ee7ab55\" (UID: \"92293c6c-8d0e-432f-b90b-a2659ee7ab55\") "
Jul 13 19:14:03.335361 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:14:03.335259    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/92293c6c-8d0e-432f-b90b-a2659ee7ab55-home\") pod \"92293c6c-8d0e-432f-b90b-a2659ee7ab55\" (UID: \"92293c6c-8d0e-432f-b90b-a2659ee7ab55\") "
Jul 13 19:14:03.335361 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:14:03.335285    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6zgbs\" (UniqueName: \"kubernetes.io/projected/92293c6c-8d0e-432f-b90b-a2659ee7ab55-kube-api-access-6zgbs\") pod \"92293c6c-8d0e-432f-b90b-a2659ee7ab55\" (UID: \"92293c6c-8d0e-432f-b90b-a2659ee7ab55\") "
Jul 13 19:14:03.335361 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:14:03.335312    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/92293c6c-8d0e-432f-b90b-a2659ee7ab55-kserve-provision-location\") pod \"92293c6c-8d0e-432f-b90b-a2659ee7ab55\" (UID: \"92293c6c-8d0e-432f-b90b-a2659ee7ab55\") "
Jul 13 19:14:03.335738 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:14:03.335420    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92293c6c-8d0e-432f-b90b-a2659ee7ab55-model-cache" (OuterVolumeSpecName: "model-cache") pod "92293c6c-8d0e-432f-b90b-a2659ee7ab55" (UID: "92293c6c-8d0e-432f-b90b-a2659ee7ab55"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:14:03.335738 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:14:03.335643    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/92293c6c-8d0e-432f-b90b-a2659ee7ab55-model-cache\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 13 19:14:03.336904 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:14:03.336875    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92293c6c-8d0e-432f-b90b-a2659ee7ab55-home" (OuterVolumeSpecName: "home") pod "92293c6c-8d0e-432f-b90b-a2659ee7ab55" (UID: "92293c6c-8d0e-432f-b90b-a2659ee7ab55"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:14:03.338210 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:14:03.338168    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92293c6c-8d0e-432f-b90b-a2659ee7ab55-dshm" (OuterVolumeSpecName: "dshm") pod "92293c6c-8d0e-432f-b90b-a2659ee7ab55" (UID: "92293c6c-8d0e-432f-b90b-a2659ee7ab55"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:14:03.338326 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:14:03.338216    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92293c6c-8d0e-432f-b90b-a2659ee7ab55-kube-api-access-6zgbs" (OuterVolumeSpecName: "kube-api-access-6zgbs") pod "92293c6c-8d0e-432f-b90b-a2659ee7ab55" (UID: "92293c6c-8d0e-432f-b90b-a2659ee7ab55"). InnerVolumeSpecName "kube-api-access-6zgbs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 19:14:03.338467 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:14:03.338429    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92293c6c-8d0e-432f-b90b-a2659ee7ab55-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "92293c6c-8d0e-432f-b90b-a2659ee7ab55" (UID: "92293c6c-8d0e-432f-b90b-a2659ee7ab55"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 19:14:03.354543 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:14:03.354496    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92293c6c-8d0e-432f-b90b-a2659ee7ab55-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "92293c6c-8d0e-432f-b90b-a2659ee7ab55" (UID: "92293c6c-8d0e-432f-b90b-a2659ee7ab55"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:14:03.355165 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:14:03.355133    2565 generic.go:356] "Generic (PLEG): container finished" podID="92293c6c-8d0e-432f-b90b-a2659ee7ab55" containerID="f3394c970fa757d42c5689a51de1a7f4cbc470caba3d6775de91d3f4e9fd9fdd" exitCode=0
Jul 13 19:14:03.355312 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:14:03.355211    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-ccd54d6d4-cgwk9"
Jul 13 19:14:03.355312 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:14:03.355213    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-ccd54d6d4-cgwk9" event={"ID":"92293c6c-8d0e-432f-b90b-a2659ee7ab55","Type":"ContainerDied","Data":"f3394c970fa757d42c5689a51de1a7f4cbc470caba3d6775de91d3f4e9fd9fdd"}
Jul 13 19:14:03.355312 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:14:03.355287    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-ccd54d6d4-cgwk9" event={"ID":"92293c6c-8d0e-432f-b90b-a2659ee7ab55","Type":"ContainerDied","Data":"a041b0d580135e2baf251fe0b5d7a86504d122b4231a20a4c20811cc0342aa0e"}
Jul 13 19:14:03.355487 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:14:03.355321    2565 scope.go:117] "RemoveContainer" containerID="f3394c970fa757d42c5689a51de1a7f4cbc470caba3d6775de91d3f4e9fd9fdd"
Jul 13 19:14:03.367290 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:14:03.367267    2565 scope.go:117] "RemoveContainer" containerID="1f0c41664078b53ae7771c4d2f2eb3bbaa7cbc0cac6e329b607f9ff7d4d43225"
Jul 13 19:14:03.378331 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:14:03.378305    2565 scope.go:117] "RemoveContainer" containerID="f3394c970fa757d42c5689a51de1a7f4cbc470caba3d6775de91d3f4e9fd9fdd"
Jul 13 19:14:03.378686 ip-10-0-128-215 kubenswrapper[2565]: E0713 19:14:03.378666    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3394c970fa757d42c5689a51de1a7f4cbc470caba3d6775de91d3f4e9fd9fdd\": container with ID starting with f3394c970fa757d42c5689a51de1a7f4cbc470caba3d6775de91d3f4e9fd9fdd not found: ID does not exist" containerID="f3394c970fa757d42c5689a51de1a7f4cbc470caba3d6775de91d3f4e9fd9fdd"
Jul 13 19:14:03.378798 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:14:03.378697    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3394c970fa757d42c5689a51de1a7f4cbc470caba3d6775de91d3f4e9fd9fdd"} err="failed to get container status \"f3394c970fa757d42c5689a51de1a7f4cbc470caba3d6775de91d3f4e9fd9fdd\": rpc error: code = NotFound desc = could not find container \"f3394c970fa757d42c5689a51de1a7f4cbc470caba3d6775de91d3f4e9fd9fdd\": container with ID starting with f3394c970fa757d42c5689a51de1a7f4cbc470caba3d6775de91d3f4e9fd9fdd not found: ID does not exist"
Jul 13 19:14:03.378798 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:14:03.378734    2565 scope.go:117] "RemoveContainer" containerID="1f0c41664078b53ae7771c4d2f2eb3bbaa7cbc0cac6e329b607f9ff7d4d43225"
Jul 13 19:14:03.379017 ip-10-0-128-215 kubenswrapper[2565]: E0713 19:14:03.378989    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f0c41664078b53ae7771c4d2f2eb3bbaa7cbc0cac6e329b607f9ff7d4d43225\": container with ID starting with 1f0c41664078b53ae7771c4d2f2eb3bbaa7cbc0cac6e329b607f9ff7d4d43225 not found: ID does not exist" containerID="1f0c41664078b53ae7771c4d2f2eb3bbaa7cbc0cac6e329b607f9ff7d4d43225"
Jul 13 19:14:03.379107 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:14:03.379026    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f0c41664078b53ae7771c4d2f2eb3bbaa7cbc0cac6e329b607f9ff7d4d43225"} err="failed to get container status \"1f0c41664078b53ae7771c4d2f2eb3bbaa7cbc0cac6e329b607f9ff7d4d43225\": rpc error: code = NotFound desc = could not find container \"1f0c41664078b53ae7771c4d2f2eb3bbaa7cbc0cac6e329b607f9ff7d4d43225\": container with ID starting with 1f0c41664078b53ae7771c4d2f2eb3bbaa7cbc0cac6e329b607f9ff7d4d43225 not found: ID does not exist"
Jul 13 19:14:03.411358 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:14:03.411302    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92293c6c-8d0e-432f-b90b-a2659ee7ab55-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "92293c6c-8d0e-432f-b90b-a2659ee7ab55" (UID: "92293c6c-8d0e-432f-b90b-a2659ee7ab55"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:14:03.437024 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:14:03.436977    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/92293c6c-8d0e-432f-b90b-a2659ee7ab55-home\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 13 19:14:03.437024 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:14:03.437019    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6zgbs\" (UniqueName: \"kubernetes.io/projected/92293c6c-8d0e-432f-b90b-a2659ee7ab55-kube-api-access-6zgbs\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 13 19:14:03.437581 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:14:03.437035    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/92293c6c-8d0e-432f-b90b-a2659ee7ab55-kserve-provision-location\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 13 19:14:03.437581 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:14:03.437051    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/92293c6c-8d0e-432f-b90b-a2659ee7ab55-dshm\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 13 19:14:03.437581 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:14:03.437067    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/92293c6c-8d0e-432f-b90b-a2659ee7ab55-tls-certs\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 13 19:14:03.437581 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:14:03.437082    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/92293c6c-8d0e-432f-b90b-a2659ee7ab55-tmp-dir\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 13 19:14:03.687474 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:14:03.687439    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/auth-disabled-test-kserve-ccd54d6d4-cgwk9"]
Jul 13 19:14:03.690909 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:14:03.690839    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/auth-disabled-test-kserve-ccd54d6d4-cgwk9"]
Jul 13 19:14:04.050138 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:14:04.050101    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="92293c6c-8d0e-432f-b90b-a2659ee7ab55" path="/var/lib/kubelet/pods/92293c6c-8d0e-432f-b90b-a2659ee7ab55/volumes"
Jul 13 19:14:08.387647 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:14:08.387609    2565 generic.go:356] "Generic (PLEG): container finished" podID="87a48a7a-6d63-43f0-8bab-7e29c1a6979a" containerID="4eb92dc57c51186beb2db648074a6552379a677ccbaf1c815310aa15999081b0" exitCode=137
Jul 13 19:14:08.388154 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:14:08.387690    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-8g97n" event={"ID":"87a48a7a-6d63-43f0-8bab-7e29c1a6979a","Type":"ContainerDied","Data":"4eb92dc57c51186beb2db648074a6552379a677ccbaf1c815310aa15999081b0"}
Jul 13 19:14:08.388154 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:14:08.387792    2565 scope.go:117] "RemoveContainer" containerID="dd3948ff3e8e5c8ecb3d67eb4faf9886831fcaafebef990c0de293d4aa95f089"
Jul 13 19:14:08.388289 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:14:08.388219    2565 scope.go:117] "RemoveContainer" containerID="4eb92dc57c51186beb2db648074a6552379a677ccbaf1c815310aa15999081b0"
Jul 13 19:14:08.388450 ip-10-0-128-215 kubenswrapper[2565]: E0713 19:14:08.388416    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 10s restarting failed container=istio-proxy pod=openshift-ai-inference-openshift-default-655f7c58b4-8g97n_openshift-ingress(87a48a7a-6d63-43f0-8bab-7e29c1a6979a)\"" pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-8g97n" podUID="87a48a7a-6d63-43f0-8bab-7e29c1a6979a"
Jul 13 19:14:09.315261 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:14:09.315222    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79c7c55747-lfmdf" podUID="b52ab7ba-8876-4a5a-afe8-c4ed84200beb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": dial tcp 10.134.0.47:8000: connect: connection refused"
Jul 13 19:14:17.546656 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:14:17.546610    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-8g97n"
Jul 13 19:14:17.547224 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:14:17.547203    2565 scope.go:117] "RemoveContainer" containerID="4eb92dc57c51186beb2db648074a6552379a677ccbaf1c815310aa15999081b0"
Jul 13 19:14:17.548495 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:14:17.548462    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 13 19:14:17.548578 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:14:17.548557    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 13 19:14:17.548622 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:14:17.548601    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 13 19:14:18.445950 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:14:18.445910    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-8g97n" event={"ID":"87a48a7a-6d63-43f0-8bab-7e29c1a6979a","Type":"ContainerStarted","Data":"14cce5f13f412f1a18a382725fd2aa4c08c9b379cc58f31dd5652248e1e07e2e"}
Jul 13 19:14:18.523775 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:14:18.523730    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-8g97n"
Jul 13 19:14:18.524993 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:14:18.524963    2565 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-655f7c58b4-8g97n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body=
Jul 13 19:14:18.525122 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:14:18.525034    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-8g97n" podUID="87a48a7a-6d63-43f0-8bab-7e29c1a6979a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused"
Jul 13 19:14:19.314848 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:14:19.314797    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79c7c55747-lfmdf" podUID="b52ab7ba-8876-4a5a-afe8-c4ed84200beb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": dial tcp 10.134.0.47:8000: connect: connection refused"
Jul 13 19:14:19.524242 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:14:19.524200    2565 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-655f7c58b4-8g97n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body=
Jul 13 19:14:19.524580 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:14:19.524545    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-8g97n" podUID="87a48a7a-6d63-43f0-8bab-7e29c1a6979a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused"
Jul 13 19:14:20.523742 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:14:20.523682    2565 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-655f7c58b4-8g97n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body=
Jul 13 19:14:20.524340 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:14:20.524306    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-8g97n" podUID="87a48a7a-6d63-43f0-8bab-7e29c1a6979a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused"
Jul 13 19:14:21.524053 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:14:21.524013    2565 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-655f7c58b4-8g97n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body=
Jul 13 19:14:21.524557 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:14:21.524082    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-8g97n" podUID="87a48a7a-6d63-43f0-8bab-7e29c1a6979a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused"
Jul 13 19:14:22.523147 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:14:22.523107    2565 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-655f7c58b4-8g97n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body=
Jul 13 19:14:22.523451 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:14:22.523418    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-8g97n" podUID="87a48a7a-6d63-43f0-8bab-7e29c1a6979a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused"
Jul 13 19:14:23.523276 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:14:23.523232    2565 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-655f7c58b4-8g97n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body=
Jul 13 19:14:23.523663 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:14:23.523307    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-8g97n" podUID="87a48a7a-6d63-43f0-8bab-7e29c1a6979a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused"
Jul 13 19:14:25.110474 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:14:25.110426    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-8g97n"
Jul 13 19:14:25.110997 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:14:25.110972    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-8g97n"
Jul 13 19:14:25.112263 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:14:25.112240    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-655f7c58b4-8g97n"
Jul 13 19:14:29.315204 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:14:29.315158    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79c7c55747-lfmdf" podUID="b52ab7ba-8876-4a5a-afe8-c4ed84200beb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": dial tcp 10.134.0.47:8000: connect: connection refused"
Jul 13 19:14:39.315029 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:14:39.314978    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79c7c55747-lfmdf" podUID="b52ab7ba-8876-4a5a-afe8-c4ed84200beb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": dial tcp 10.134.0.47:8000: connect: connection refused"
Jul 13 19:14:49.331687 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:14:49.331654    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79c7c55747-lfmdf"
Jul 13 19:14:49.345512 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:14:49.345475    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79c7c55747-lfmdf"
Jul 13 19:15:13.993763 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:15:13.993727    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2bscc_2f0655bd-4a09-4314-8c1b-e30191eaec0e/ovn-acl-logging/0.log"
Jul 13 19:15:13.997665 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:15:13.997631    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2bscc_2f0655bd-4a09-4314-8c1b-e30191eaec0e/ovn-acl-logging/0.log"
Jul 13 19:15:17.407212 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:15:17.407171    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79c7c55747-lfmdf"]
Jul 13 19:15:32.431899 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:15:32.431828    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79c7c55747-lfmdf" podUID="b52ab7ba-8876-4a5a-afe8-c4ed84200beb" containerName="main" containerID="cri-o://16e1decef4d0e439378e5a2de4be9d838215e6f40b3760a06839ecf7177d4f21" gracePeriod=45
Jul 13 19:15:33.093057 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:15:33.093031    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79c7c55747-lfmdf"
Jul 13 19:15:33.204306 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:15:33.204222    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b52ab7ba-8876-4a5a-afe8-c4ed84200beb-tls-certs\") pod \"b52ab7ba-8876-4a5a-afe8-c4ed84200beb\" (UID: \"b52ab7ba-8876-4a5a-afe8-c4ed84200beb\") "
Jul 13 19:15:33.204306 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:15:33.204267    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b52ab7ba-8876-4a5a-afe8-c4ed84200beb-home\") pod \"b52ab7ba-8876-4a5a-afe8-c4ed84200beb\" (UID: \"b52ab7ba-8876-4a5a-afe8-c4ed84200beb\") "
Jul 13 19:15:33.204306 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:15:33.204297    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b52ab7ba-8876-4a5a-afe8-c4ed84200beb-cabundle-cert\") pod \"b52ab7ba-8876-4a5a-afe8-c4ed84200beb\" (UID: \"b52ab7ba-8876-4a5a-afe8-c4ed84200beb\") "
Jul 13 19:15:33.204536 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:15:33.204325    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b52ab7ba-8876-4a5a-afe8-c4ed84200beb-kserve-provision-location\") pod \"b52ab7ba-8876-4a5a-afe8-c4ed84200beb\" (UID: \"b52ab7ba-8876-4a5a-afe8-c4ed84200beb\") "
Jul 13 19:15:33.204536 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:15:33.204380    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b52ab7ba-8876-4a5a-afe8-c4ed84200beb-model-cache\") pod \"b52ab7ba-8876-4a5a-afe8-c4ed84200beb\" (UID: \"b52ab7ba-8876-4a5a-afe8-c4ed84200beb\") "
Jul 13 19:15:33.204536 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:15:33.204481    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zwxgn\" (UniqueName: \"kubernetes.io/projected/b52ab7ba-8876-4a5a-afe8-c4ed84200beb-kube-api-access-zwxgn\") pod \"b52ab7ba-8876-4a5a-afe8-c4ed84200beb\" (UID: \"b52ab7ba-8876-4a5a-afe8-c4ed84200beb\") "
Jul 13 19:15:33.204536 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:15:33.204519    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b52ab7ba-8876-4a5a-afe8-c4ed84200beb-tmp-dir\") pod \"b52ab7ba-8876-4a5a-afe8-c4ed84200beb\" (UID: \"b52ab7ba-8876-4a5a-afe8-c4ed84200beb\") "
Jul 13 19:15:33.204777 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:15:33.204547    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b52ab7ba-8876-4a5a-afe8-c4ed84200beb-dshm\") pod \"b52ab7ba-8876-4a5a-afe8-c4ed84200beb\" (UID: \"b52ab7ba-8876-4a5a-afe8-c4ed84200beb\") "
Jul 13 19:15:33.204777 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:15:33.204674    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b52ab7ba-8876-4a5a-afe8-c4ed84200beb-model-cache" (OuterVolumeSpecName: "model-cache") pod "b52ab7ba-8876-4a5a-afe8-c4ed84200beb" (UID: "b52ab7ba-8876-4a5a-afe8-c4ed84200beb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:15:33.204777 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:15:33.204757    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b52ab7ba-8876-4a5a-afe8-c4ed84200beb-home" (OuterVolumeSpecName: "home") pod "b52ab7ba-8876-4a5a-afe8-c4ed84200beb" (UID: "b52ab7ba-8876-4a5a-afe8-c4ed84200beb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:15:33.204777 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:15:33.204771    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b52ab7ba-8876-4a5a-afe8-c4ed84200beb-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "b52ab7ba-8876-4a5a-afe8-c4ed84200beb" (UID: "b52ab7ba-8876-4a5a-afe8-c4ed84200beb"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 19:15:33.205006 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:15:33.204954    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b52ab7ba-8876-4a5a-afe8-c4ed84200beb-home\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 13 19:15:33.205006 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:15:33.204974    2565 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b52ab7ba-8876-4a5a-afe8-c4ed84200beb-cabundle-cert\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 13 19:15:33.205006 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:15:33.204989    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b52ab7ba-8876-4a5a-afe8-c4ed84200beb-model-cache\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 13 19:15:33.206783 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:15:33.206667    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b52ab7ba-8876-4a5a-afe8-c4ed84200beb-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b52ab7ba-8876-4a5a-afe8-c4ed84200beb" (UID: "b52ab7ba-8876-4a5a-afe8-c4ed84200beb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 19:15:33.206918 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:15:33.206820    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b52ab7ba-8876-4a5a-afe8-c4ed84200beb-kube-api-access-zwxgn" (OuterVolumeSpecName: "kube-api-access-zwxgn") pod "b52ab7ba-8876-4a5a-afe8-c4ed84200beb" (UID: "b52ab7ba-8876-4a5a-afe8-c4ed84200beb"). InnerVolumeSpecName "kube-api-access-zwxgn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 19:15:33.207351 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:15:33.207329    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b52ab7ba-8876-4a5a-afe8-c4ed84200beb-dshm" (OuterVolumeSpecName: "dshm") pod "b52ab7ba-8876-4a5a-afe8-c4ed84200beb" (UID: "b52ab7ba-8876-4a5a-afe8-c4ed84200beb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:15:33.215670 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:15:33.215645    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b52ab7ba-8876-4a5a-afe8-c4ed84200beb-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b52ab7ba-8876-4a5a-afe8-c4ed84200beb" (UID: "b52ab7ba-8876-4a5a-afe8-c4ed84200beb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:15:33.260901 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:15:33.260874    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b52ab7ba-8876-4a5a-afe8-c4ed84200beb-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b52ab7ba-8876-4a5a-afe8-c4ed84200beb" (UID: "b52ab7ba-8876-4a5a-afe8-c4ed84200beb"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:15:33.306085 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:15:33.306065    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zwxgn\" (UniqueName: \"kubernetes.io/projected/b52ab7ba-8876-4a5a-afe8-c4ed84200beb-kube-api-access-zwxgn\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 13 19:15:33.306085 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:15:33.306085    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b52ab7ba-8876-4a5a-afe8-c4ed84200beb-tmp-dir\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 13 19:15:33.306204 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:15:33.306096    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b52ab7ba-8876-4a5a-afe8-c4ed84200beb-dshm\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 13 19:15:33.306204 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:15:33.306104    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b52ab7ba-8876-4a5a-afe8-c4ed84200beb-tls-certs\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 13 19:15:33.306204 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:15:33.306113    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b52ab7ba-8876-4a5a-afe8-c4ed84200beb-kserve-provision-location\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 13 19:15:33.872584 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:15:33.872554    2565 generic.go:356] "Generic (PLEG): container finished" podID="b52ab7ba-8876-4a5a-afe8-c4ed84200beb" containerID="16e1decef4d0e439378e5a2de4be9d838215e6f40b3760a06839ecf7177d4f21" exitCode=0
Jul 13 19:15:33.874027 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:15:33.872608    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79c7c55747-lfmdf"
Jul 13 19:15:33.874027 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:15:33.872609    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79c7c55747-lfmdf" event={"ID":"b52ab7ba-8876-4a5a-afe8-c4ed84200beb","Type":"ContainerDied","Data":"16e1decef4d0e439378e5a2de4be9d838215e6f40b3760a06839ecf7177d4f21"}
Jul 13 19:15:33.874027 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:15:33.872755    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79c7c55747-lfmdf" event={"ID":"b52ab7ba-8876-4a5a-afe8-c4ed84200beb","Type":"ContainerDied","Data":"d76913aa056ed6a264c62627256563fd13c453af5ac72831093defcc07de46b5"}
Jul 13 19:15:33.874027 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:15:33.872788    2565 scope.go:117] "RemoveContainer" containerID="16e1decef4d0e439378e5a2de4be9d838215e6f40b3760a06839ecf7177d4f21"
Jul 13 19:15:33.882112 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:15:33.882095    2565 scope.go:117] "RemoveContainer" containerID="5f94646180cfd466d27c560dc17166dffb03a8819f21ae5802c79750cd1fa7dc"
Jul 13 19:15:33.890258 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:15:33.890245    2565 scope.go:117] "RemoveContainer" containerID="16e1decef4d0e439378e5a2de4be9d838215e6f40b3760a06839ecf7177d4f21"
Jul 13 19:15:33.890500 ip-10-0-128-215 kubenswrapper[2565]: E0713 19:15:33.890484    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16e1decef4d0e439378e5a2de4be9d838215e6f40b3760a06839ecf7177d4f21\": container with ID starting with 16e1decef4d0e439378e5a2de4be9d838215e6f40b3760a06839ecf7177d4f21 not found: ID does not exist" containerID="16e1decef4d0e439378e5a2de4be9d838215e6f40b3760a06839ecf7177d4f21"
Jul 13 19:15:33.890540 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:15:33.890508    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16e1decef4d0e439378e5a2de4be9d838215e6f40b3760a06839ecf7177d4f21"} err="failed to get container status \"16e1decef4d0e439378e5a2de4be9d838215e6f40b3760a06839ecf7177d4f21\": rpc error: code = NotFound desc = could not find container \"16e1decef4d0e439378e5a2de4be9d838215e6f40b3760a06839ecf7177d4f21\": container with ID starting with 16e1decef4d0e439378e5a2de4be9d838215e6f40b3760a06839ecf7177d4f21 not found: ID does not exist"
Jul 13 19:15:33.890540 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:15:33.890526    2565 scope.go:117] "RemoveContainer" containerID="5f94646180cfd466d27c560dc17166dffb03a8819f21ae5802c79750cd1fa7dc"
Jul 13 19:15:33.890790 ip-10-0-128-215 kubenswrapper[2565]: E0713 19:15:33.890771    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f94646180cfd466d27c560dc17166dffb03a8819f21ae5802c79750cd1fa7dc\": container with ID starting with 5f94646180cfd466d27c560dc17166dffb03a8819f21ae5802c79750cd1fa7dc not found: ID does not exist" containerID="5f94646180cfd466d27c560dc17166dffb03a8819f21ae5802c79750cd1fa7dc"
Jul 13 19:15:33.890839 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:15:33.890797    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f94646180cfd466d27c560dc17166dffb03a8819f21ae5802c79750cd1fa7dc"} err="failed to get container status \"5f94646180cfd466d27c560dc17166dffb03a8819f21ae5802c79750cd1fa7dc\": rpc error: code = NotFound desc = could not find container \"5f94646180cfd466d27c560dc17166dffb03a8819f21ae5802c79750cd1fa7dc\": container with ID starting with 5f94646180cfd466d27c560dc17166dffb03a8819f21ae5802c79750cd1fa7dc not found: ID does not exist"
Jul 13 19:15:33.898925 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:15:33.898902    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79c7c55747-lfmdf"]
Jul 13 19:15:33.904129 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:15:33.904107    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-79c7c55747-lfmdf"]
Jul 13 19:15:34.048692 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:15:34.048655    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b52ab7ba-8876-4a5a-afe8-c4ed84200beb" path="/var/lib/kubelet/pods/b52ab7ba-8876-4a5a-afe8-c4ed84200beb/volumes"
Jul 13 19:15:48.082514 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:15:48.082477    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-794bb95d4d-s6j8f"]
Jul 13 19:15:48.083368 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:15:48.082941    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="92293c6c-8d0e-432f-b90b-a2659ee7ab55" containerName="storage-initializer"
Jul 13 19:15:48.083368 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:15:48.082954    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="92293c6c-8d0e-432f-b90b-a2659ee7ab55" containerName="storage-initializer"
Jul 13 19:15:48.083368 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:15:48.082968    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b52ab7ba-8876-4a5a-afe8-c4ed84200beb" containerName="storage-initializer"
Jul 13 19:15:48.083368 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:15:48.082973    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="b52ab7ba-8876-4a5a-afe8-c4ed84200beb" containerName="storage-initializer"
Jul 13 19:15:48.083368 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:15:48.082979    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b52ab7ba-8876-4a5a-afe8-c4ed84200beb" containerName="main"
Jul 13 19:15:48.083368 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:15:48.082986    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="b52ab7ba-8876-4a5a-afe8-c4ed84200beb" containerName="main"
Jul 13 19:15:48.083368 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:15:48.082992    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="92293c6c-8d0e-432f-b90b-a2659ee7ab55" containerName="main"
Jul 13 19:15:48.083368 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:15:48.082999    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="92293c6c-8d0e-432f-b90b-a2659ee7ab55" containerName="main"
Jul 13 19:15:48.083368 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:15:48.083077    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="92293c6c-8d0e-432f-b90b-a2659ee7ab55" containerName="main"
Jul 13 19:15:48.083368 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:15:48.083096    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="b52ab7ba-8876-4a5a-afe8-c4ed84200beb" containerName="main"
Jul 13 19:15:48.086253 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:15:48.086233    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-794bb95d4d-s6j8f"
Jul 13 19:15:48.089094 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:15:48.089067    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 13 19:15:48.089241 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:15:48.089169    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 13 19:15:48.089320 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:15:48.089287    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 13 19:15:48.089320 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:15:48.089307    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 13 19:15:48.089533 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:15:48.089516    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-gv2sm\""
Jul 13 19:15:48.090403 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:15:48.090370    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv77ff2528d3e9b4972cd9335229fce9f0-kserve-self-signed-certs\""
Jul 13 19:15:48.096923 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:15:48.096896    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-794bb95d4d-s6j8f"]
Jul 13 19:15:48.239120 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:15:48.239071    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/487a99c1-2fec-4486-951b-f7cf7acd96b5-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-794bb95d4d-s6j8f\" (UID: \"487a99c1-2fec-4486-951b-f7cf7acd96b5\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-794bb95d4d-s6j8f"
Jul 13 19:15:48.239120 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:15:48.239118    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnbgt\" (UniqueName: \"kubernetes.io/projected/487a99c1-2fec-4486-951b-f7cf7acd96b5-kube-api-access-cnbgt\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-794bb95d4d-s6j8f\" (UID: \"487a99c1-2fec-4486-951b-f7cf7acd96b5\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-794bb95d4d-s6j8f"
Jul 13 19:15:48.239358 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:15:48.239223    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/487a99c1-2fec-4486-951b-f7cf7acd96b5-home\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-794bb95d4d-s6j8f\" (UID: \"487a99c1-2fec-4486-951b-f7cf7acd96b5\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-794bb95d4d-s6j8f"
Jul 13 19:15:48.239358 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:15:48.239259    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/487a99c1-2fec-4486-951b-f7cf7acd96b5-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-794bb95d4d-s6j8f\" (UID: \"487a99c1-2fec-4486-951b-f7cf7acd96b5\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-794bb95d4d-s6j8f"
Jul 13 19:15:48.239358 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:15:48.239339    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/487a99c1-2fec-4486-951b-f7cf7acd96b5-dshm\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-794bb95d4d-s6j8f\" (UID: \"487a99c1-2fec-4486-951b-f7cf7acd96b5\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-794bb95d4d-s6j8f"
Jul 13 19:15:48.239489 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:15:48.239366    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/487a99c1-2fec-4486-951b-f7cf7acd96b5-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-794bb95d4d-s6j8f\" (UID: \"487a99c1-2fec-4486-951b-f7cf7acd96b5\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-794bb95d4d-s6j8f"
Jul 13 19:15:48.239489 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:15:48.239402    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/487a99c1-2fec-4486-951b-f7cf7acd96b5-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-794bb95d4d-s6j8f\" (UID: \"487a99c1-2fec-4486-951b-f7cf7acd96b5\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-794bb95d4d-s6j8f"
Jul 13 19:15:48.239489 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:15:48.239474    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/487a99c1-2fec-4486-951b-f7cf7acd96b5-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-794bb95d4d-s6j8f\" (UID: \"487a99c1-2fec-4486-951b-f7cf7acd96b5\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-794bb95d4d-s6j8f"
Jul 13 19:15:48.340500 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:15:48.340391    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/487a99c1-2fec-4486-951b-f7cf7acd96b5-dshm\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-794bb95d4d-s6j8f\" (UID: \"487a99c1-2fec-4486-951b-f7cf7acd96b5\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-794bb95d4d-s6j8f"
Jul 13 19:15:48.340500 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:15:48.340436    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/487a99c1-2fec-4486-951b-f7cf7acd96b5-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-794bb95d4d-s6j8f\" (UID: \"487a99c1-2fec-4486-951b-f7cf7acd96b5\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-794bb95d4d-s6j8f"
Jul 13 19:15:48.340500 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:15:48.340464    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/487a99c1-2fec-4486-951b-f7cf7acd96b5-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-794bb95d4d-s6j8f\" (UID: \"487a99c1-2fec-4486-951b-f7cf7acd96b5\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-794bb95d4d-s6j8f"
Jul 13 19:15:48.340500 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:15:48.340502    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/487a99c1-2fec-4486-951b-f7cf7acd96b5-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-794bb95d4d-s6j8f\" (UID: \"487a99c1-2fec-4486-951b-f7cf7acd96b5\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-794bb95d4d-s6j8f"
Jul 13 19:15:48.340866 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:15:48.340529    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/487a99c1-2fec-4486-951b-f7cf7acd96b5-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-794bb95d4d-s6j8f\" (UID: \"487a99c1-2fec-4486-951b-f7cf7acd96b5\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-794bb95d4d-s6j8f"
Jul 13 19:15:48.340866 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:15:48.340544    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnbgt\" (UniqueName: \"kubernetes.io/projected/487a99c1-2fec-4486-951b-f7cf7acd96b5-kube-api-access-cnbgt\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-794bb95d4d-s6j8f\" (UID: \"487a99c1-2fec-4486-951b-f7cf7acd96b5\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-794bb95d4d-s6j8f"
Jul 13 19:15:48.340866 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:15:48.340582    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/487a99c1-2fec-4486-951b-f7cf7acd96b5-home\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-794bb95d4d-s6j8f\" (UID: \"487a99c1-2fec-4486-951b-f7cf7acd96b5\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-794bb95d4d-s6j8f"
Jul 13 19:15:48.340866 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:15:48.340606    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/487a99c1-2fec-4486-951b-f7cf7acd96b5-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-794bb95d4d-s6j8f\" (UID: \"487a99c1-2fec-4486-951b-f7cf7acd96b5\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-794bb95d4d-s6j8f"
Jul 13 19:15:48.341083 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:15:48.340991    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/487a99c1-2fec-4486-951b-f7cf7acd96b5-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-794bb95d4d-s6j8f\" (UID: \"487a99c1-2fec-4486-951b-f7cf7acd96b5\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-794bb95d4d-s6j8f"
Jul 13 19:15:48.341142 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:15:48.341105    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/487a99c1-2fec-4486-951b-f7cf7acd96b5-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-794bb95d4d-s6j8f\" (UID: \"487a99c1-2fec-4486-951b-f7cf7acd96b5\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-794bb95d4d-s6j8f"
Jul 13 19:15:48.341186 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:15:48.341144    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/487a99c1-2fec-4486-951b-f7cf7acd96b5-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-794bb95d4d-s6j8f\" (UID: \"487a99c1-2fec-4486-951b-f7cf7acd96b5\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-794bb95d4d-s6j8f"
Jul 13 19:15:48.341222 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:15:48.341187    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/487a99c1-2fec-4486-951b-f7cf7acd96b5-home\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-794bb95d4d-s6j8f\" (UID: \"487a99c1-2fec-4486-951b-f7cf7acd96b5\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-794bb95d4d-s6j8f"
Jul 13 19:15:48.341803 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:15:48.341777    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/487a99c1-2fec-4486-951b-f7cf7acd96b5-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-794bb95d4d-s6j8f\" (UID: \"487a99c1-2fec-4486-951b-f7cf7acd96b5\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-794bb95d4d-s6j8f"
Jul 13 19:15:48.343400 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:15:48.343377    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/487a99c1-2fec-4486-951b-f7cf7acd96b5-dshm\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-794bb95d4d-s6j8f\" (UID: \"487a99c1-2fec-4486-951b-f7cf7acd96b5\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-794bb95d4d-s6j8f"
Jul 13 19:15:48.343537 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:15:48.343407    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/487a99c1-2fec-4486-951b-f7cf7acd96b5-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-794bb95d4d-s6j8f\" (UID: \"487a99c1-2fec-4486-951b-f7cf7acd96b5\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-794bb95d4d-s6j8f"
Jul 13 19:15:48.348529 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:15:48.348498    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnbgt\" (UniqueName: \"kubernetes.io/projected/487a99c1-2fec-4486-951b-f7cf7acd96b5-kube-api-access-cnbgt\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-794bb95d4d-s6j8f\" (UID: \"487a99c1-2fec-4486-951b-f7cf7acd96b5\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-794bb95d4d-s6j8f"
Jul 13 19:15:48.397646 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:15:48.397605    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-794bb95d4d-s6j8f"
Jul 13 19:15:48.560419 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:15:48.560386    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-794bb95d4d-s6j8f"]
Jul 13 19:15:48.561902 ip-10-0-128-215 kubenswrapper[2565]: W0713 19:15:48.561867    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod487a99c1_2fec_4486_951b_f7cf7acd96b5.slice/crio-3aca08497b94f18999fd0eebb59ab9a4f312644cafb99b87e32f1f07db2d27bd WatchSource:0}: Error finding container 3aca08497b94f18999fd0eebb59ab9a4f312644cafb99b87e32f1f07db2d27bd: Status 404 returned error can't find the container with id 3aca08497b94f18999fd0eebb59ab9a4f312644cafb99b87e32f1f07db2d27bd
Jul 13 19:15:48.948857 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:15:48.948741    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-794bb95d4d-s6j8f" event={"ID":"487a99c1-2fec-4486-951b-f7cf7acd96b5","Type":"ContainerStarted","Data":"531b07264cfb9f1327499a336eb9a9eb1e3a1037df482d47d478a9fb9cf4f8d8"}
Jul 13 19:15:48.948857 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:15:48.948812    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-794bb95d4d-s6j8f" event={"ID":"487a99c1-2fec-4486-951b-f7cf7acd96b5","Type":"ContainerStarted","Data":"3aca08497b94f18999fd0eebb59ab9a4f312644cafb99b87e32f1f07db2d27bd"}
Jul 13 19:15:52.969919 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:15:52.969879    2565 generic.go:356] "Generic (PLEG): container finished" podID="487a99c1-2fec-4486-951b-f7cf7acd96b5" containerID="531b07264cfb9f1327499a336eb9a9eb1e3a1037df482d47d478a9fb9cf4f8d8" exitCode=0
Jul 13 19:15:52.970351 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:15:52.969955    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-794bb95d4d-s6j8f" event={"ID":"487a99c1-2fec-4486-951b-f7cf7acd96b5","Type":"ContainerDied","Data":"531b07264cfb9f1327499a336eb9a9eb1e3a1037df482d47d478a9fb9cf4f8d8"}
Jul 13 19:15:53.979531 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:15:53.979498    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-794bb95d4d-s6j8f" event={"ID":"487a99c1-2fec-4486-951b-f7cf7acd96b5","Type":"ContainerStarted","Data":"2967d2a1419f6bccadc4ccd3b79466fd9169cca408697555e7612b98776575e7"}
Jul 13 19:15:54.021099 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:15:54.016195    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-794bb95d4d-s6j8f" podStartSLOduration=2.356928108 podStartE2EDuration="6.016173552s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.659245444s" isStatefulPod=false podCreationTimestamp="2026-07-13 19:15:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 19:15:54.016006767 +0000 UTC m=+1540.541140209" watchObservedRunningTime="2026-07-13 19:15:54.016173552 +0000 UTC m=+1540.541306970"
Jul 13 19:15:54.398075 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:15:54.398043    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-794bb95d4d-s6j8f"
Jul 13 19:15:58.397929 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:15:58.397892    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-794bb95d4d-s6j8f"
Jul 13 19:15:58.399502 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:15:58.399469    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-794bb95d4d-s6j8f" podUID="487a99c1-2fec-4486-951b-f7cf7acd96b5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": dial tcp 10.134.0.48:8000: connect: connection refused"
Jul 13 19:16:08.398268 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:16:08.398214    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-794bb95d4d-s6j8f" podUID="487a99c1-2fec-4486-951b-f7cf7acd96b5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": dial tcp 10.134.0.48:8000: connect: connection refused"
Jul 13 19:16:18.398111 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:16:18.398066    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-794bb95d4d-s6j8f" podUID="487a99c1-2fec-4486-951b-f7cf7acd96b5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": dial tcp 10.134.0.48:8000: connect: connection refused"
Jul 13 19:16:28.398208 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:16:28.398156    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-794bb95d4d-s6j8f" podUID="487a99c1-2fec-4486-951b-f7cf7acd96b5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": dial tcp 10.134.0.48:8000: connect: connection refused"
Jul 13 19:16:38.398386 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:16:38.398337    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-794bb95d4d-s6j8f" podUID="487a99c1-2fec-4486-951b-f7cf7acd96b5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": dial tcp 10.134.0.48:8000: connect: connection refused"
Jul 13 19:16:48.398354 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:16:48.398308    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-794bb95d4d-s6j8f" podUID="487a99c1-2fec-4486-951b-f7cf7acd96b5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": dial tcp 10.134.0.48:8000: connect: connection refused"
Jul 13 19:16:56.301770 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:16:56.301737    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5db6d9f47djrd27"]
Jul 13 19:16:56.310032 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:16:56.310006    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5db6d9f47djrd27"
Jul 13 19:16:56.313268 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:16:56.313240    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisva690bbc929faec8bc98c767f16c003c1-kserve-self-signed-certs\""
Jul 13 19:16:56.320422 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:16:56.320399    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5db6d9f47djrd27"]
Jul 13 19:16:56.375683 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:16:56.375654    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69cdf14a-bdab-4416-84fa-db9747852ccc-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5db6d9f47djrd27\" (UID: \"69cdf14a-bdab-4416-84fa-db9747852ccc\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5db6d9f47djrd27"
Jul 13 19:16:56.375864 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:16:56.375691    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/69cdf14a-bdab-4416-84fa-db9747852ccc-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5db6d9f47djrd27\" (UID: \"69cdf14a-bdab-4416-84fa-db9747852ccc\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5db6d9f47djrd27"
Jul 13 19:16:56.375864 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:16:56.375747    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8nl2k\" (UniqueName: \"kubernetes.io/projected/69cdf14a-bdab-4416-84fa-db9747852ccc-kube-api-access-8nl2k\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5db6d9f47djrd27\" (UID: \"69cdf14a-bdab-4416-84fa-db9747852ccc\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5db6d9f47djrd27"
Jul 13 19:16:56.375864 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:16:56.375797    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69cdf14a-bdab-4416-84fa-db9747852ccc-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5db6d9f47djrd27\" (UID: \"69cdf14a-bdab-4416-84fa-db9747852ccc\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5db6d9f47djrd27"
Jul 13 19:16:56.375981 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:16:56.375925    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69cdf14a-bdab-4416-84fa-db9747852ccc-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5db6d9f47djrd27\" (UID: \"69cdf14a-bdab-4416-84fa-db9747852ccc\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5db6d9f47djrd27"
Jul 13 19:16:56.375981 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:16:56.375968    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/69cdf14a-bdab-4416-84fa-db9747852ccc-home\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5db6d9f47djrd27\" (UID: \"69cdf14a-bdab-4416-84fa-db9747852ccc\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5db6d9f47djrd27"
Jul 13 19:16:56.376063 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:16:56.375993    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/69cdf14a-bdab-4416-84fa-db9747852ccc-dshm\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5db6d9f47djrd27\" (UID: \"69cdf14a-bdab-4416-84fa-db9747852ccc\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5db6d9f47djrd27"
Jul 13 19:16:56.477152 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:16:56.477116    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69cdf14a-bdab-4416-84fa-db9747852ccc-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5db6d9f47djrd27\" (UID: \"69cdf14a-bdab-4416-84fa-db9747852ccc\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5db6d9f47djrd27"
Jul 13 19:16:56.477363 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:16:56.477173    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/69cdf14a-bdab-4416-84fa-db9747852ccc-home\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5db6d9f47djrd27\" (UID: \"69cdf14a-bdab-4416-84fa-db9747852ccc\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5db6d9f47djrd27"
Jul 13 19:16:56.477363 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:16:56.477203    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/69cdf14a-bdab-4416-84fa-db9747852ccc-dshm\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5db6d9f47djrd27\" (UID: \"69cdf14a-bdab-4416-84fa-db9747852ccc\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5db6d9f47djrd27"
Jul 13 19:16:56.477363 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:16:56.477292    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69cdf14a-bdab-4416-84fa-db9747852ccc-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5db6d9f47djrd27\" (UID: \"69cdf14a-bdab-4416-84fa-db9747852ccc\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5db6d9f47djrd27"
Jul 13 19:16:56.477363 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:16:56.477322    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/69cdf14a-bdab-4416-84fa-db9747852ccc-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5db6d9f47djrd27\" (UID: \"69cdf14a-bdab-4416-84fa-db9747852ccc\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5db6d9f47djrd27"
Jul 13 19:16:56.477363 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:16:56.477350    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8nl2k\" (UniqueName: \"kubernetes.io/projected/69cdf14a-bdab-4416-84fa-db9747852ccc-kube-api-access-8nl2k\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5db6d9f47djrd27\" (UID: \"69cdf14a-bdab-4416-84fa-db9747852ccc\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5db6d9f47djrd27"
Jul 13 19:16:56.477651 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:16:56.477407    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69cdf14a-bdab-4416-84fa-db9747852ccc-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5db6d9f47djrd27\" (UID: \"69cdf14a-bdab-4416-84fa-db9747852ccc\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5db6d9f47djrd27"
Jul 13 19:16:56.477651 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:16:56.477582    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69cdf14a-bdab-4416-84fa-db9747852ccc-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5db6d9f47djrd27\" (UID: \"69cdf14a-bdab-4416-84fa-db9747852ccc\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5db6d9f47djrd27"
Jul 13 19:16:56.477885 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:16:56.477860    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69cdf14a-bdab-4416-84fa-db9747852ccc-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5db6d9f47djrd27\" (UID: \"69cdf14a-bdab-4416-84fa-db9747852ccc\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5db6d9f47djrd27"
Jul 13 19:16:56.477969 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:16:56.477901    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/69cdf14a-bdab-4416-84fa-db9747852ccc-home\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5db6d9f47djrd27\" (UID: \"69cdf14a-bdab-4416-84fa-db9747852ccc\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5db6d9f47djrd27"
Jul 13 19:16:56.478101 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:16:56.478077    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/69cdf14a-bdab-4416-84fa-db9747852ccc-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5db6d9f47djrd27\" (UID: \"69cdf14a-bdab-4416-84fa-db9747852ccc\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5db6d9f47djrd27"
Jul 13 19:16:56.480332 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:16:56.480310    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/69cdf14a-bdab-4416-84fa-db9747852ccc-dshm\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5db6d9f47djrd27\" (UID: \"69cdf14a-bdab-4416-84fa-db9747852ccc\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5db6d9f47djrd27"
Jul 13 19:16:56.480422 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:16:56.480342    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69cdf14a-bdab-4416-84fa-db9747852ccc-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5db6d9f47djrd27\" (UID: \"69cdf14a-bdab-4416-84fa-db9747852ccc\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5db6d9f47djrd27"
Jul 13 19:16:56.486548 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:16:56.486522    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8nl2k\" (UniqueName: \"kubernetes.io/projected/69cdf14a-bdab-4416-84fa-db9747852ccc-kube-api-access-8nl2k\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5db6d9f47djrd27\" (UID: \"69cdf14a-bdab-4416-84fa-db9747852ccc\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5db6d9f47djrd27"
Jul 13 19:16:56.622781 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:16:56.622676    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5db6d9f47djrd27"
Jul 13 19:16:56.765886 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:16:56.765827    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5db6d9f47djrd27"]
Jul 13 19:16:56.768582 ip-10-0-128-215 kubenswrapper[2565]: W0713 19:16:56.768539    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69cdf14a_bdab_4416_84fa_db9747852ccc.slice/crio-49a8df61c627da2a4ad972726b86b84b5ab1a534f912bb5ae69b1f3ab431a559 WatchSource:0}: Error finding container 49a8df61c627da2a4ad972726b86b84b5ab1a534f912bb5ae69b1f3ab431a559: Status 404 returned error can't find the container with id 49a8df61c627da2a4ad972726b86b84b5ab1a534f912bb5ae69b1f3ab431a559
Jul 13 19:16:57.336928 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:16:57.336889    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5db6d9f47djrd27" event={"ID":"69cdf14a-bdab-4416-84fa-db9747852ccc","Type":"ContainerStarted","Data":"b9dcb4f8c2dc5c9e151372dcf8a7e5db07a4169a536805ca97e0a7101886f130"}
Jul 13 19:16:57.337388 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:16:57.336956    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5db6d9f47djrd27" event={"ID":"69cdf14a-bdab-4416-84fa-db9747852ccc","Type":"ContainerStarted","Data":"49a8df61c627da2a4ad972726b86b84b5ab1a534f912bb5ae69b1f3ab431a559"}
Jul 13 19:16:58.398492 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:16:58.398439    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-794bb95d4d-s6j8f" podUID="487a99c1-2fec-4486-951b-f7cf7acd96b5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": dial tcp 10.134.0.48:8000: connect: connection refused"
Jul 13 19:17:00.355891 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:17:00.355859    2565 generic.go:356] "Generic (PLEG): container finished" podID="69cdf14a-bdab-4416-84fa-db9747852ccc" containerID="b9dcb4f8c2dc5c9e151372dcf8a7e5db07a4169a536805ca97e0a7101886f130" exitCode=0
Jul 13 19:17:00.356311 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:17:00.355928    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5db6d9f47djrd27" event={"ID":"69cdf14a-bdab-4416-84fa-db9747852ccc","Type":"ContainerDied","Data":"b9dcb4f8c2dc5c9e151372dcf8a7e5db07a4169a536805ca97e0a7101886f130"}
Jul 13 19:17:01.366970 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:17:01.366915    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5db6d9f47djrd27" event={"ID":"69cdf14a-bdab-4416-84fa-db9747852ccc","Type":"ContainerStarted","Data":"f9c00dd062014fb16cc858abf6974d0686a9779e4ccf122c384b411ec1c9c6a7"}
Jul 13 19:17:01.393553 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:17:01.393504    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5db6d9f47djrd27" podStartSLOduration=2.114936361 podStartE2EDuration="5.393489889s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.278553528s" isStatefulPod=false podCreationTimestamp="2026-07-13 19:16:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 19:17:01.391770108 +0000 UTC m=+1607.916903533" watchObservedRunningTime="2026-07-13 19:17:01.393489889 +0000 UTC m=+1607.918623304"
Jul 13 19:17:01.623372 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:17:01.623277    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5db6d9f47djrd27"
Jul 13 19:17:06.623587 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:17:06.623488    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5db6d9f47djrd27"
Jul 13 19:17:06.625426 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:17:06.625387    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5db6d9f47djrd27" podUID="69cdf14a-bdab-4416-84fa-db9747852ccc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": dial tcp 10.134.0.49:8000: connect: connection refused"
Jul 13 19:17:08.408798 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:17:08.408764    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-794bb95d4d-s6j8f"
Jul 13 19:17:08.417152 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:17:08.417122    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-794bb95d4d-s6j8f"
Jul 13 19:17:09.770773 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:17:09.770736    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-794bb95d4d-s6j8f"]
Jul 13 19:17:16.623642 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:17:16.623594    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5db6d9f47djrd27" podUID="69cdf14a-bdab-4416-84fa-db9747852ccc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": dial tcp 10.134.0.49:8000: connect: connection refused"
Jul 13 19:17:25.441453 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:17:25.441411    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-794bb95d4d-s6j8f" podUID="487a99c1-2fec-4486-951b-f7cf7acd96b5" containerName="main" containerID="cri-o://2967d2a1419f6bccadc4ccd3b79466fd9169cca408697555e7612b98776575e7" gracePeriod=45
Jul 13 19:17:25.993549 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:17:25.993524    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-794bb95d4d-s6j8f"
Jul 13 19:17:26.165033 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:17:26.164941    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/487a99c1-2fec-4486-951b-f7cf7acd96b5-model-cache\") pod \"487a99c1-2fec-4486-951b-f7cf7acd96b5\" (UID: \"487a99c1-2fec-4486-951b-f7cf7acd96b5\") "
Jul 13 19:17:26.165033 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:17:26.165024    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/487a99c1-2fec-4486-951b-f7cf7acd96b5-cabundle-cert\") pod \"487a99c1-2fec-4486-951b-f7cf7acd96b5\" (UID: \"487a99c1-2fec-4486-951b-f7cf7acd96b5\") "
Jul 13 19:17:26.165263 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:17:26.165066    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/487a99c1-2fec-4486-951b-f7cf7acd96b5-kserve-provision-location\") pod \"487a99c1-2fec-4486-951b-f7cf7acd96b5\" (UID: \"487a99c1-2fec-4486-951b-f7cf7acd96b5\") "
Jul 13 19:17:26.165263 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:17:26.165094    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/487a99c1-2fec-4486-951b-f7cf7acd96b5-dshm\") pod \"487a99c1-2fec-4486-951b-f7cf7acd96b5\" (UID: \"487a99c1-2fec-4486-951b-f7cf7acd96b5\") "
Jul 13 19:17:26.165263 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:17:26.165227    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/487a99c1-2fec-4486-951b-f7cf7acd96b5-model-cache" (OuterVolumeSpecName: "model-cache") pod "487a99c1-2fec-4486-951b-f7cf7acd96b5" (UID: "487a99c1-2fec-4486-951b-f7cf7acd96b5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:17:26.165453 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:17:26.165260    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/487a99c1-2fec-4486-951b-f7cf7acd96b5-home\") pod \"487a99c1-2fec-4486-951b-f7cf7acd96b5\" (UID: \"487a99c1-2fec-4486-951b-f7cf7acd96b5\") "
Jul 13 19:17:26.165453 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:17:26.165349    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cnbgt\" (UniqueName: \"kubernetes.io/projected/487a99c1-2fec-4486-951b-f7cf7acd96b5-kube-api-access-cnbgt\") pod \"487a99c1-2fec-4486-951b-f7cf7acd96b5\" (UID: \"487a99c1-2fec-4486-951b-f7cf7acd96b5\") "
Jul 13 19:17:26.165453 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:17:26.165377    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/487a99c1-2fec-4486-951b-f7cf7acd96b5-tls-certs\") pod \"487a99c1-2fec-4486-951b-f7cf7acd96b5\" (UID: \"487a99c1-2fec-4486-951b-f7cf7acd96b5\") "
Jul 13 19:17:26.165453 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:17:26.165418    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/487a99c1-2fec-4486-951b-f7cf7acd96b5-tmp-dir\") pod \"487a99c1-2fec-4486-951b-f7cf7acd96b5\" (UID: \"487a99c1-2fec-4486-951b-f7cf7acd96b5\") "
Jul 13 19:17:26.165453 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:17:26.165433    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/487a99c1-2fec-4486-951b-f7cf7acd96b5-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "487a99c1-2fec-4486-951b-f7cf7acd96b5" (UID: "487a99c1-2fec-4486-951b-f7cf7acd96b5"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 13 19:17:26.165741 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:17:26.165693    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/487a99c1-2fec-4486-951b-f7cf7acd96b5-home" (OuterVolumeSpecName: "home") pod "487a99c1-2fec-4486-951b-f7cf7acd96b5" (UID: "487a99c1-2fec-4486-951b-f7cf7acd96b5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:17:26.165848 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:17:26.165826    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/487a99c1-2fec-4486-951b-f7cf7acd96b5-model-cache\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 13 19:17:26.166013 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:17:26.165999    2565 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/487a99c1-2fec-4486-951b-f7cf7acd96b5-cabundle-cert\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 13 19:17:26.166251 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:17:26.166210    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/487a99c1-2fec-4486-951b-f7cf7acd96b5-home\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 13 19:17:26.166370 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:17:26.165999    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/487a99c1-2fec-4486-951b-f7cf7acd96b5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "487a99c1-2fec-4486-951b-f7cf7acd96b5" (UID: "487a99c1-2fec-4486-951b-f7cf7acd96b5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:17:26.167657 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:17:26.167631    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/487a99c1-2fec-4486-951b-f7cf7acd96b5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "487a99c1-2fec-4486-951b-f7cf7acd96b5" (UID: "487a99c1-2fec-4486-951b-f7cf7acd96b5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 19:17:26.167759 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:17:26.167736    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/487a99c1-2fec-4486-951b-f7cf7acd96b5-dshm" (OuterVolumeSpecName: "dshm") pod "487a99c1-2fec-4486-951b-f7cf7acd96b5" (UID: "487a99c1-2fec-4486-951b-f7cf7acd96b5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:17:26.167883 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:17:26.167863    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/487a99c1-2fec-4486-951b-f7cf7acd96b5-kube-api-access-cnbgt" (OuterVolumeSpecName: "kube-api-access-cnbgt") pod "487a99c1-2fec-4486-951b-f7cf7acd96b5" (UID: "487a99c1-2fec-4486-951b-f7cf7acd96b5"). InnerVolumeSpecName "kube-api-access-cnbgt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 19:17:26.229389 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:17:26.229350    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/487a99c1-2fec-4486-951b-f7cf7acd96b5-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "487a99c1-2fec-4486-951b-f7cf7acd96b5" (UID: "487a99c1-2fec-4486-951b-f7cf7acd96b5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:17:26.267027 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:17:26.267000    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/487a99c1-2fec-4486-951b-f7cf7acd96b5-dshm\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 13 19:17:26.267027 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:17:26.267025    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cnbgt\" (UniqueName: \"kubernetes.io/projected/487a99c1-2fec-4486-951b-f7cf7acd96b5-kube-api-access-cnbgt\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 13 19:17:26.267178 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:17:26.267035    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/487a99c1-2fec-4486-951b-f7cf7acd96b5-tls-certs\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 13 19:17:26.267178 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:17:26.267045    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/487a99c1-2fec-4486-951b-f7cf7acd96b5-tmp-dir\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 13 19:17:26.267178 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:17:26.267053    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/487a99c1-2fec-4486-951b-f7cf7acd96b5-kserve-provision-location\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 13 19:17:26.498555 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:17:26.498446    2565 generic.go:356] "Generic (PLEG): container finished" podID="487a99c1-2fec-4486-951b-f7cf7acd96b5" containerID="2967d2a1419f6bccadc4ccd3b79466fd9169cca408697555e7612b98776575e7" exitCode=0
Jul 13 19:17:26.498555 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:17:26.498512    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-794bb95d4d-s6j8f"
Jul 13 19:17:26.498555 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:17:26.498529    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-794bb95d4d-s6j8f" event={"ID":"487a99c1-2fec-4486-951b-f7cf7acd96b5","Type":"ContainerDied","Data":"2967d2a1419f6bccadc4ccd3b79466fd9169cca408697555e7612b98776575e7"}
Jul 13 19:17:26.499201 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:17:26.498593    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-794bb95d4d-s6j8f" event={"ID":"487a99c1-2fec-4486-951b-f7cf7acd96b5","Type":"ContainerDied","Data":"3aca08497b94f18999fd0eebb59ab9a4f312644cafb99b87e32f1f07db2d27bd"}
Jul 13 19:17:26.499201 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:17:26.498623    2565 scope.go:117] "RemoveContainer" containerID="2967d2a1419f6bccadc4ccd3b79466fd9169cca408697555e7612b98776575e7"
Jul 13 19:17:26.509458 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:17:26.509441    2565 scope.go:117] "RemoveContainer" containerID="531b07264cfb9f1327499a336eb9a9eb1e3a1037df482d47d478a9fb9cf4f8d8"
Jul 13 19:17:26.517530 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:17:26.517508    2565 scope.go:117] "RemoveContainer" containerID="2967d2a1419f6bccadc4ccd3b79466fd9169cca408697555e7612b98776575e7"
Jul 13 19:17:26.517931 ip-10-0-128-215 kubenswrapper[2565]: E0713 19:17:26.517912    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2967d2a1419f6bccadc4ccd3b79466fd9169cca408697555e7612b98776575e7\": container with ID starting with 2967d2a1419f6bccadc4ccd3b79466fd9169cca408697555e7612b98776575e7 not found: ID does not exist" containerID="2967d2a1419f6bccadc4ccd3b79466fd9169cca408697555e7612b98776575e7"
Jul 13 19:17:26.518004 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:17:26.517940    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2967d2a1419f6bccadc4ccd3b79466fd9169cca408697555e7612b98776575e7"} err="failed to get container status \"2967d2a1419f6bccadc4ccd3b79466fd9169cca408697555e7612b98776575e7\": rpc error: code = NotFound desc = could not find container \"2967d2a1419f6bccadc4ccd3b79466fd9169cca408697555e7612b98776575e7\": container with ID starting with 2967d2a1419f6bccadc4ccd3b79466fd9169cca408697555e7612b98776575e7 not found: ID does not exist"
Jul 13 19:17:26.518004 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:17:26.517956    2565 scope.go:117] "RemoveContainer" containerID="531b07264cfb9f1327499a336eb9a9eb1e3a1037df482d47d478a9fb9cf4f8d8"
Jul 13 19:17:26.518222 ip-10-0-128-215 kubenswrapper[2565]: E0713 19:17:26.518206    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"531b07264cfb9f1327499a336eb9a9eb1e3a1037df482d47d478a9fb9cf4f8d8\": container with ID starting with 531b07264cfb9f1327499a336eb9a9eb1e3a1037df482d47d478a9fb9cf4f8d8 not found: ID does not exist" containerID="531b07264cfb9f1327499a336eb9a9eb1e3a1037df482d47d478a9fb9cf4f8d8"
Jul 13 19:17:26.518266 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:17:26.518226    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"531b07264cfb9f1327499a336eb9a9eb1e3a1037df482d47d478a9fb9cf4f8d8"} err="failed to get container status \"531b07264cfb9f1327499a336eb9a9eb1e3a1037df482d47d478a9fb9cf4f8d8\": rpc error: code = NotFound desc = could not find container \"531b07264cfb9f1327499a336eb9a9eb1e3a1037df482d47d478a9fb9cf4f8d8\": container with ID starting with 531b07264cfb9f1327499a336eb9a9eb1e3a1037df482d47d478a9fb9cf4f8d8 not found: ID does not exist"
Jul 13 19:17:26.520115 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:17:26.520095    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-794bb95d4d-s6j8f"]
Jul 13 19:17:26.526068 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:17:26.526046    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-794bb95d4d-s6j8f"]
Jul 13 19:17:26.623948 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:17:26.623916    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5db6d9f47djrd27" podUID="69cdf14a-bdab-4416-84fa-db9747852ccc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": dial tcp 10.134.0.49:8000: connect: connection refused"
Jul 13 19:17:28.047899 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:17:28.047870    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="487a99c1-2fec-4486-951b-f7cf7acd96b5" path="/var/lib/kubelet/pods/487a99c1-2fec-4486-951b-f7cf7acd96b5/volumes"
Jul 13 19:17:36.623819 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:17:36.623770    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5db6d9f47djrd27" podUID="69cdf14a-bdab-4416-84fa-db9747852ccc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": dial tcp 10.134.0.49:8000: connect: connection refused"
Jul 13 19:17:46.623476 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:17:46.623429    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5db6d9f47djrd27" podUID="69cdf14a-bdab-4416-84fa-db9747852ccc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": dial tcp 10.134.0.49:8000: connect: connection refused"
Jul 13 19:17:51.793874 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:17:51.793833    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dmww7f"]
Jul 13 19:17:51.794434 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:17:51.794415    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="487a99c1-2fec-4486-951b-f7cf7acd96b5" containerName="main"
Jul 13 19:17:51.794487 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:17:51.794439    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="487a99c1-2fec-4486-951b-f7cf7acd96b5" containerName="main"
Jul 13 19:17:51.794487 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:17:51.794480    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="487a99c1-2fec-4486-951b-f7cf7acd96b5" containerName="storage-initializer"
Jul 13 19:17:51.794565 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:17:51.794490    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="487a99c1-2fec-4486-951b-f7cf7acd96b5" containerName="storage-initializer"
Jul 13 19:17:51.794615 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:17:51.794603    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="487a99c1-2fec-4486-951b-f7cf7acd96b5" containerName="main"
Jul 13 19:17:51.797300 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:17:51.797282    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dmww7f"
Jul 13 19:17:51.799964 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:17:51.799943    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv8bf079eb6eda4debfb4ef5bb7817824c-kserve-self-signed-certs\""
Jul 13 19:17:51.808588 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:17:51.808566    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dmww7f"]
Jul 13 19:17:51.892397 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:17:51.892358    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/634b78c4-751c-4880-9f6c-1ea69b737987-home\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dmww7f\" (UID: \"634b78c4-751c-4880-9f6c-1ea69b737987\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dmww7f"
Jul 13 19:17:51.892397 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:17:51.892393    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/634b78c4-751c-4880-9f6c-1ea69b737987-model-cache\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dmww7f\" (UID: \"634b78c4-751c-4880-9f6c-1ea69b737987\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dmww7f"
Jul 13 19:17:51.892618 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:17:51.892426    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ghrx\" (UniqueName: \"kubernetes.io/projected/634b78c4-751c-4880-9f6c-1ea69b737987-kube-api-access-4ghrx\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dmww7f\" (UID: \"634b78c4-751c-4880-9f6c-1ea69b737987\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dmww7f"
Jul 13 19:17:51.892618 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:17:51.892527    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/634b78c4-751c-4880-9f6c-1ea69b737987-dshm\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dmww7f\" (UID: \"634b78c4-751c-4880-9f6c-1ea69b737987\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dmww7f"
Jul 13 19:17:51.892618 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:17:51.892602    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/634b78c4-751c-4880-9f6c-1ea69b737987-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dmww7f\" (UID: \"634b78c4-751c-4880-9f6c-1ea69b737987\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dmww7f"
Jul 13 19:17:51.892744 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:17:51.892626    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/634b78c4-751c-4880-9f6c-1ea69b737987-tls-certs\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dmww7f\" (UID: \"634b78c4-751c-4880-9f6c-1ea69b737987\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dmww7f"
Jul 13 19:17:51.993789 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:17:51.993741    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/634b78c4-751c-4880-9f6c-1ea69b737987-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dmww7f\" (UID: \"634b78c4-751c-4880-9f6c-1ea69b737987\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dmww7f"
Jul 13 19:17:51.993789 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:17:51.993792    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/634b78c4-751c-4880-9f6c-1ea69b737987-tls-certs\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dmww7f\" (UID: \"634b78c4-751c-4880-9f6c-1ea69b737987\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dmww7f"
Jul 13 19:17:51.994044 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:17:51.993880    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/634b78c4-751c-4880-9f6c-1ea69b737987-home\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dmww7f\" (UID: \"634b78c4-751c-4880-9f6c-1ea69b737987\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dmww7f"
Jul 13 19:17:51.994044 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:17:51.993910    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/634b78c4-751c-4880-9f6c-1ea69b737987-model-cache\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dmww7f\" (UID: \"634b78c4-751c-4880-9f6c-1ea69b737987\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dmww7f"
Jul 13 19:17:51.994044 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:17:51.993965    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ghrx\" (UniqueName: \"kubernetes.io/projected/634b78c4-751c-4880-9f6c-1ea69b737987-kube-api-access-4ghrx\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dmww7f\" (UID: \"634b78c4-751c-4880-9f6c-1ea69b737987\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dmww7f"
Jul 13 19:17:51.994044 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:17:51.994027    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/634b78c4-751c-4880-9f6c-1ea69b737987-dshm\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dmww7f\" (UID: \"634b78c4-751c-4880-9f6c-1ea69b737987\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dmww7f"
Jul 13 19:17:51.994251 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:17:51.994151    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/634b78c4-751c-4880-9f6c-1ea69b737987-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dmww7f\" (UID: \"634b78c4-751c-4880-9f6c-1ea69b737987\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dmww7f"
Jul 13 19:17:51.994410 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:17:51.994390    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/634b78c4-751c-4880-9f6c-1ea69b737987-model-cache\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dmww7f\" (UID: \"634b78c4-751c-4880-9f6c-1ea69b737987\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dmww7f"
Jul 13 19:17:51.994834 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:17:51.994812    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/634b78c4-751c-4880-9f6c-1ea69b737987-home\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dmww7f\" (UID: \"634b78c4-751c-4880-9f6c-1ea69b737987\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dmww7f"
Jul 13 19:17:51.996695 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:17:51.996665    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/634b78c4-751c-4880-9f6c-1ea69b737987-dshm\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dmww7f\" (UID: \"634b78c4-751c-4880-9f6c-1ea69b737987\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dmww7f"
Jul 13 19:17:51.997128 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:17:51.997106    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/634b78c4-751c-4880-9f6c-1ea69b737987-tls-certs\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dmww7f\" (UID: \"634b78c4-751c-4880-9f6c-1ea69b737987\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dmww7f"
Jul 13 19:17:52.002422 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:17:52.002397    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ghrx\" (UniqueName: \"kubernetes.io/projected/634b78c4-751c-4880-9f6c-1ea69b737987-kube-api-access-4ghrx\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dmww7f\" (UID: \"634b78c4-751c-4880-9f6c-1ea69b737987\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dmww7f"
Jul 13 19:17:56.623083 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:17:56.623033    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5db6d9f47djrd27" podUID="69cdf14a-bdab-4416-84fa-db9747852ccc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": dial tcp 10.134.0.49:8000: connect: connection refused"
Jul 13 19:18:06.623597 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:18:06.623558    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5db6d9f47djrd27" podUID="69cdf14a-bdab-4416-84fa-db9747852ccc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": dial tcp 10.134.0.49:8000: connect: connection refused"
Jul 13 19:18:16.624210 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:18:16.624165    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5db6d9f47djrd27" podUID="69cdf14a-bdab-4416-84fa-db9747852ccc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": dial tcp 10.134.0.49:8000: connect: connection refused"
Jul 13 19:18:26.623541 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:18:26.623500    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5db6d9f47djrd27" podUID="69cdf14a-bdab-4416-84fa-db9747852ccc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": dial tcp 10.134.0.49:8000: connect: connection refused"
Jul 13 19:18:36.623246 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:18:36.623145    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5db6d9f47djrd27" podUID="69cdf14a-bdab-4416-84fa-db9747852ccc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": dial tcp 10.134.0.49:8000: connect: connection refused"
Jul 13 19:18:46.623908 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:18:46.623855    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5db6d9f47djrd27" podUID="69cdf14a-bdab-4416-84fa-db9747852ccc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": dial tcp 10.134.0.49:8000: connect: connection refused"
Jul 13 19:18:56.633197 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:18:56.633167    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5db6d9f47djrd27"
Jul 13 19:18:56.641137 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:18:56.641110    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5db6d9f47djrd27"
Jul 13 19:19:07.111147 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:19:07.111108    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5db6d9f47djrd27"]
Jul 13 19:19:22.135870 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:19:22.135814    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5db6d9f47djrd27" podUID="69cdf14a-bdab-4416-84fa-db9747852ccc" containerName="main" containerID="cri-o://f9c00dd062014fb16cc858abf6974d0686a9779e4ccf122c384b411ec1c9c6a7" gracePeriod=45
Jul 13 19:19:22.634627 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:19:22.634584    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5db6d9f47djrd27" podUID="69cdf14a-bdab-4416-84fa-db9747852ccc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": dial tcp 10.134.0.49:8000: connect: connection refused"
Jul 13 19:19:22.891296 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:19:22.891239    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5db6d9f47djrd27"
Jul 13 19:19:22.959995 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:19:22.959968    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69cdf14a-bdab-4416-84fa-db9747852ccc-tmp-dir\") pod \"69cdf14a-bdab-4416-84fa-db9747852ccc\" (UID: \"69cdf14a-bdab-4416-84fa-db9747852ccc\") "
Jul 13 19:19:22.959995 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:19:22.959997    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/69cdf14a-bdab-4416-84fa-db9747852ccc-kserve-provision-location\") pod \"69cdf14a-bdab-4416-84fa-db9747852ccc\" (UID: \"69cdf14a-bdab-4416-84fa-db9747852ccc\") "
Jul 13 19:19:22.960208 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:19:22.960023    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8nl2k\" (UniqueName: \"kubernetes.io/projected/69cdf14a-bdab-4416-84fa-db9747852ccc-kube-api-access-8nl2k\") pod \"69cdf14a-bdab-4416-84fa-db9747852ccc\" (UID: \"69cdf14a-bdab-4416-84fa-db9747852ccc\") "
Jul 13 19:19:22.960208 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:19:22.960061    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/69cdf14a-bdab-4416-84fa-db9747852ccc-home\") pod \"69cdf14a-bdab-4416-84fa-db9747852ccc\" (UID: \"69cdf14a-bdab-4416-84fa-db9747852ccc\") "
Jul 13 19:19:22.960208 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:19:22.960110    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69cdf14a-bdab-4416-84fa-db9747852ccc-model-cache\") pod \"69cdf14a-bdab-4416-84fa-db9747852ccc\" (UID: \"69cdf14a-bdab-4416-84fa-db9747852ccc\") "
Jul 13 19:19:22.960208 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:19:22.960134    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/69cdf14a-bdab-4416-84fa-db9747852ccc-dshm\") pod \"69cdf14a-bdab-4416-84fa-db9747852ccc\" (UID: \"69cdf14a-bdab-4416-84fa-db9747852ccc\") "
Jul 13 19:19:22.960425 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:19:22.960263    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69cdf14a-bdab-4416-84fa-db9747852ccc-tls-certs\") pod \"69cdf14a-bdab-4416-84fa-db9747852ccc\" (UID: \"69cdf14a-bdab-4416-84fa-db9747852ccc\") "
Jul 13 19:19:22.960577 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:19:22.960541    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69cdf14a-bdab-4416-84fa-db9747852ccc-model-cache" (OuterVolumeSpecName: "model-cache") pod "69cdf14a-bdab-4416-84fa-db9747852ccc" (UID: "69cdf14a-bdab-4416-84fa-db9747852ccc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:19:22.960998 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:19:22.960973    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69cdf14a-bdab-4416-84fa-db9747852ccc-home" (OuterVolumeSpecName: "home") pod "69cdf14a-bdab-4416-84fa-db9747852ccc" (UID: "69cdf14a-bdab-4416-84fa-db9747852ccc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:19:22.962460 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:19:22.962421    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69cdf14a-bdab-4416-84fa-db9747852ccc-kube-api-access-8nl2k" (OuterVolumeSpecName: "kube-api-access-8nl2k") pod "69cdf14a-bdab-4416-84fa-db9747852ccc" (UID: "69cdf14a-bdab-4416-84fa-db9747852ccc"). InnerVolumeSpecName "kube-api-access-8nl2k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 19:19:22.962566 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:19:22.962474    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69cdf14a-bdab-4416-84fa-db9747852ccc-dshm" (OuterVolumeSpecName: "dshm") pod "69cdf14a-bdab-4416-84fa-db9747852ccc" (UID: "69cdf14a-bdab-4416-84fa-db9747852ccc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:19:22.962740 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:19:22.962724    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/69cdf14a-bdab-4416-84fa-db9747852ccc-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "69cdf14a-bdab-4416-84fa-db9747852ccc" (UID: "69cdf14a-bdab-4416-84fa-db9747852ccc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 19:19:22.979122 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:19:22.979095    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69cdf14a-bdab-4416-84fa-db9747852ccc-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "69cdf14a-bdab-4416-84fa-db9747852ccc" (UID: "69cdf14a-bdab-4416-84fa-db9747852ccc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:19:23.021194 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:19:23.021170    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69cdf14a-bdab-4416-84fa-db9747852ccc-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "69cdf14a-bdab-4416-84fa-db9747852ccc" (UID: "69cdf14a-bdab-4416-84fa-db9747852ccc"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:19:23.061961 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:19:23.061935    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69cdf14a-bdab-4416-84fa-db9747852ccc-tls-certs\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 13 19:19:23.061961 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:19:23.061957    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69cdf14a-bdab-4416-84fa-db9747852ccc-tmp-dir\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 13 19:19:23.062091 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:19:23.061966    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/69cdf14a-bdab-4416-84fa-db9747852ccc-kserve-provision-location\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 13 19:19:23.062091 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:19:23.061975    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8nl2k\" (UniqueName: \"kubernetes.io/projected/69cdf14a-bdab-4416-84fa-db9747852ccc-kube-api-access-8nl2k\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 13 19:19:23.062091 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:19:23.061984    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/69cdf14a-bdab-4416-84fa-db9747852ccc-home\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 13 19:19:23.062091 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:19:23.061994    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69cdf14a-bdab-4416-84fa-db9747852ccc-model-cache\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 13 19:19:23.062091 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:19:23.062002    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/69cdf14a-bdab-4416-84fa-db9747852ccc-dshm\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 13 19:19:23.097899 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:19:23.097866    2565 generic.go:356] "Generic (PLEG): container finished" podID="69cdf14a-bdab-4416-84fa-db9747852ccc" containerID="f9c00dd062014fb16cc858abf6974d0686a9779e4ccf122c384b411ec1c9c6a7" exitCode=0
Jul 13 19:19:23.098010 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:19:23.097912    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5db6d9f47djrd27" event={"ID":"69cdf14a-bdab-4416-84fa-db9747852ccc","Type":"ContainerDied","Data":"f9c00dd062014fb16cc858abf6974d0686a9779e4ccf122c384b411ec1c9c6a7"}
Jul 13 19:19:23.098010 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:19:23.097922    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5db6d9f47djrd27"
Jul 13 19:19:23.098010 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:19:23.097966    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5db6d9f47djrd27" event={"ID":"69cdf14a-bdab-4416-84fa-db9747852ccc","Type":"ContainerDied","Data":"49a8df61c627da2a4ad972726b86b84b5ab1a534f912bb5ae69b1f3ab431a559"}
Jul 13 19:19:23.098010 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:19:23.097990    2565 scope.go:117] "RemoveContainer" containerID="f9c00dd062014fb16cc858abf6974d0686a9779e4ccf122c384b411ec1c9c6a7"
Jul 13 19:19:23.107234 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:19:23.107216    2565 scope.go:117] "RemoveContainer" containerID="b9dcb4f8c2dc5c9e151372dcf8a7e5db07a4169a536805ca97e0a7101886f130"
Jul 13 19:19:23.115655 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:19:23.115634    2565 scope.go:117] "RemoveContainer" containerID="f9c00dd062014fb16cc858abf6974d0686a9779e4ccf122c384b411ec1c9c6a7"
Jul 13 19:19:23.115944 ip-10-0-128-215 kubenswrapper[2565]: E0713 19:19:23.115925    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9c00dd062014fb16cc858abf6974d0686a9779e4ccf122c384b411ec1c9c6a7\": container with ID starting with f9c00dd062014fb16cc858abf6974d0686a9779e4ccf122c384b411ec1c9c6a7 not found: ID does not exist" containerID="f9c00dd062014fb16cc858abf6974d0686a9779e4ccf122c384b411ec1c9c6a7"
Jul 13 19:19:23.116014 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:19:23.115951    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9c00dd062014fb16cc858abf6974d0686a9779e4ccf122c384b411ec1c9c6a7"} err="failed to get container status \"f9c00dd062014fb16cc858abf6974d0686a9779e4ccf122c384b411ec1c9c6a7\": rpc error: code = NotFound desc = could not find container \"f9c00dd062014fb16cc858abf6974d0686a9779e4ccf122c384b411ec1c9c6a7\": container with ID starting with f9c00dd062014fb16cc858abf6974d0686a9779e4ccf122c384b411ec1c9c6a7 not found: ID does not exist"
Jul 13 19:19:23.116014 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:19:23.115967    2565 scope.go:117] "RemoveContainer" containerID="b9dcb4f8c2dc5c9e151372dcf8a7e5db07a4169a536805ca97e0a7101886f130"
Jul 13 19:19:23.116213 ip-10-0-128-215 kubenswrapper[2565]: E0713 19:19:23.116197    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9dcb4f8c2dc5c9e151372dcf8a7e5db07a4169a536805ca97e0a7101886f130\": container with ID starting with b9dcb4f8c2dc5c9e151372dcf8a7e5db07a4169a536805ca97e0a7101886f130 not found: ID does not exist" containerID="b9dcb4f8c2dc5c9e151372dcf8a7e5db07a4169a536805ca97e0a7101886f130"
Jul 13 19:19:23.116250 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:19:23.116217    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9dcb4f8c2dc5c9e151372dcf8a7e5db07a4169a536805ca97e0a7101886f130"} err="failed to get container status \"b9dcb4f8c2dc5c9e151372dcf8a7e5db07a4169a536805ca97e0a7101886f130\": rpc error: code = NotFound desc = could not find container \"b9dcb4f8c2dc5c9e151372dcf8a7e5db07a4169a536805ca97e0a7101886f130\": container with ID starting with b9dcb4f8c2dc5c9e151372dcf8a7e5db07a4169a536805ca97e0a7101886f130 not found: ID does not exist"
Jul 13 19:19:23.119399 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:19:23.119378    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5db6d9f47djrd27"]
Jul 13 19:19:23.123842 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:19:23.123822    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5db6d9f47djrd27"]
Jul 13 19:19:24.049129 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:19:24.049096    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="69cdf14a-bdab-4416-84fa-db9747852ccc" path="/var/lib/kubelet/pods/69cdf14a-bdab-4416-84fa-db9747852ccc/volumes"
Jul 13 19:19:31.616904 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:19:31.616868    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-965ccd984-nft7b"]
Jul 13 19:19:31.617290 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:19:31.617234    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="69cdf14a-bdab-4416-84fa-db9747852ccc" containerName="storage-initializer"
Jul 13 19:19:31.617290 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:19:31.617246    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="69cdf14a-bdab-4416-84fa-db9747852ccc" containerName="storage-initializer"
Jul 13 19:19:31.617290 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:19:31.617265    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="69cdf14a-bdab-4416-84fa-db9747852ccc" containerName="main"
Jul 13 19:19:31.617290 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:19:31.617271    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="69cdf14a-bdab-4416-84fa-db9747852ccc" containerName="main"
Jul 13 19:19:31.617421 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:19:31.617336    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="69cdf14a-bdab-4416-84fa-db9747852ccc" containerName="main"
Jul 13 19:19:31.620904 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:19:31.620885    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-965ccd984-nft7b"
Jul 13 19:19:31.623765 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:19:31.623745    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 13 19:19:31.623900 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:19:31.623856    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"custom-route-timeout-test-kserve-self-signed-certs\""
Jul 13 19:19:31.640732 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:19:31.635178    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-965ccd984-nft7b"]
Jul 13 19:19:31.733300 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:19:31.733266    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f742e7c-d921-4eff-b938-d73d798b7ec8-dshm\") pod \"custom-route-timeout-test-kserve-965ccd984-nft7b\" (UID: \"9f742e7c-d921-4eff-b938-d73d798b7ec8\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-965ccd984-nft7b"
Jul 13 19:19:31.733300 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:19:31.733302    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f742e7c-d921-4eff-b938-d73d798b7ec8-model-cache\") pod \"custom-route-timeout-test-kserve-965ccd984-nft7b\" (UID: \"9f742e7c-d921-4eff-b938-d73d798b7ec8\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-965ccd984-nft7b"
Jul 13 19:19:31.733566 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:19:31.733324    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gpp2q\" (UniqueName: \"kubernetes.io/projected/9f742e7c-d921-4eff-b938-d73d798b7ec8-kube-api-access-gpp2q\") pod \"custom-route-timeout-test-kserve-965ccd984-nft7b\" (UID: \"9f742e7c-d921-4eff-b938-d73d798b7ec8\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-965ccd984-nft7b"
Jul 13 19:19:31.733566 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:19:31.733357    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f742e7c-d921-4eff-b938-d73d798b7ec8-tls-certs\") pod \"custom-route-timeout-test-kserve-965ccd984-nft7b\" (UID: \"9f742e7c-d921-4eff-b938-d73d798b7ec8\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-965ccd984-nft7b"
Jul 13 19:19:31.733566 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:19:31.733462    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9f742e7c-d921-4eff-b938-d73d798b7ec8-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-965ccd984-nft7b\" (UID: \"9f742e7c-d921-4eff-b938-d73d798b7ec8\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-965ccd984-nft7b"
Jul 13 19:19:31.733566 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:19:31.733567    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f742e7c-d921-4eff-b938-d73d798b7ec8-tmp-dir\") pod \"custom-route-timeout-test-kserve-965ccd984-nft7b\" (UID: \"9f742e7c-d921-4eff-b938-d73d798b7ec8\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-965ccd984-nft7b"
Jul 13 19:19:31.733788 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:19:31.733585    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f742e7c-d921-4eff-b938-d73d798b7ec8-home\") pod \"custom-route-timeout-test-kserve-965ccd984-nft7b\" (UID: \"9f742e7c-d921-4eff-b938-d73d798b7ec8\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-965ccd984-nft7b"
Jul 13 19:19:31.834212 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:19:31.834176    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9f742e7c-d921-4eff-b938-d73d798b7ec8-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-965ccd984-nft7b\" (UID: \"9f742e7c-d921-4eff-b938-d73d798b7ec8\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-965ccd984-nft7b"
Jul 13 19:19:31.834378 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:19:31.834278    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f742e7c-d921-4eff-b938-d73d798b7ec8-tmp-dir\") pod \"custom-route-timeout-test-kserve-965ccd984-nft7b\" (UID: \"9f742e7c-d921-4eff-b938-d73d798b7ec8\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-965ccd984-nft7b"
Jul 13 19:19:31.834378 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:19:31.834298    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f742e7c-d921-4eff-b938-d73d798b7ec8-home\") pod \"custom-route-timeout-test-kserve-965ccd984-nft7b\" (UID: \"9f742e7c-d921-4eff-b938-d73d798b7ec8\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-965ccd984-nft7b"
Jul 13 19:19:31.834378 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:19:31.834320    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f742e7c-d921-4eff-b938-d73d798b7ec8-dshm\") pod \"custom-route-timeout-test-kserve-965ccd984-nft7b\" (UID: \"9f742e7c-d921-4eff-b938-d73d798b7ec8\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-965ccd984-nft7b"
Jul 13 19:19:31.834378 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:19:31.834335    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f742e7c-d921-4eff-b938-d73d798b7ec8-model-cache\") pod \"custom-route-timeout-test-kserve-965ccd984-nft7b\" (UID: \"9f742e7c-d921-4eff-b938-d73d798b7ec8\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-965ccd984-nft7b"
Jul 13 19:19:31.834378 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:19:31.834355    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpp2q\" (UniqueName: \"kubernetes.io/projected/9f742e7c-d921-4eff-b938-d73d798b7ec8-kube-api-access-gpp2q\") pod \"custom-route-timeout-test-kserve-965ccd984-nft7b\" (UID: \"9f742e7c-d921-4eff-b938-d73d798b7ec8\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-965ccd984-nft7b"
Jul 13 19:19:31.834629 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:19:31.834382    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f742e7c-d921-4eff-b938-d73d798b7ec8-tls-certs\") pod \"custom-route-timeout-test-kserve-965ccd984-nft7b\" (UID: \"9f742e7c-d921-4eff-b938-d73d798b7ec8\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-965ccd984-nft7b"
Jul 13 19:19:31.834687 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:19:31.834642    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9f742e7c-d921-4eff-b938-d73d798b7ec8-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-965ccd984-nft7b\" (UID: \"9f742e7c-d921-4eff-b938-d73d798b7ec8\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-965ccd984-nft7b"
Jul 13 19:19:31.834773 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:19:31.834687    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f742e7c-d921-4eff-b938-d73d798b7ec8-home\") pod \"custom-route-timeout-test-kserve-965ccd984-nft7b\" (UID: \"9f742e7c-d921-4eff-b938-d73d798b7ec8\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-965ccd984-nft7b"
Jul 13 19:19:31.834826 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:19:31.834746    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f742e7c-d921-4eff-b938-d73d798b7ec8-tmp-dir\") pod \"custom-route-timeout-test-kserve-965ccd984-nft7b\" (UID: \"9f742e7c-d921-4eff-b938-d73d798b7ec8\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-965ccd984-nft7b"
Jul 13 19:19:31.834878 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:19:31.834817    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f742e7c-d921-4eff-b938-d73d798b7ec8-model-cache\") pod \"custom-route-timeout-test-kserve-965ccd984-nft7b\" (UID: \"9f742e7c-d921-4eff-b938-d73d798b7ec8\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-965ccd984-nft7b"
Jul 13 19:19:31.836747 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:19:31.836726    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f742e7c-d921-4eff-b938-d73d798b7ec8-dshm\") pod \"custom-route-timeout-test-kserve-965ccd984-nft7b\" (UID: \"9f742e7c-d921-4eff-b938-d73d798b7ec8\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-965ccd984-nft7b"
Jul 13 19:19:31.837108 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:19:31.837089    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f742e7c-d921-4eff-b938-d73d798b7ec8-tls-certs\") pod \"custom-route-timeout-test-kserve-965ccd984-nft7b\" (UID: \"9f742e7c-d921-4eff-b938-d73d798b7ec8\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-965ccd984-nft7b"
Jul 13 19:19:31.846034 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:19:31.846012    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpp2q\" (UniqueName: \"kubernetes.io/projected/9f742e7c-d921-4eff-b938-d73d798b7ec8-kube-api-access-gpp2q\") pod \"custom-route-timeout-test-kserve-965ccd984-nft7b\" (UID: \"9f742e7c-d921-4eff-b938-d73d798b7ec8\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-965ccd984-nft7b"
Jul 13 19:19:31.930839 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:19:31.930755    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-965ccd984-nft7b"
Jul 13 19:19:32.072908 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:19:32.072885    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-965ccd984-nft7b"]
Jul 13 19:19:32.074545 ip-10-0-128-215 kubenswrapper[2565]: W0713 19:19:32.074518    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f742e7c_d921_4eff_b938_d73d798b7ec8.slice/crio-063d15a7c65ebe0657688ef0298d70dd1122ab3c3ff0771e8e5a40b4665bc673 WatchSource:0}: Error finding container 063d15a7c65ebe0657688ef0298d70dd1122ab3c3ff0771e8e5a40b4665bc673: Status 404 returned error can't find the container with id 063d15a7c65ebe0657688ef0298d70dd1122ab3c3ff0771e8e5a40b4665bc673
Jul 13 19:19:32.142630 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:19:32.142602    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-965ccd984-nft7b" event={"ID":"9f742e7c-d921-4eff-b938-d73d798b7ec8","Type":"ContainerStarted","Data":"f5dc0f57f4a882fba1d322373cb51ca6970ca28b6b75a3c3abf6fdea29d6ecc6"}
Jul 13 19:19:32.142778 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:19:32.142657    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-965ccd984-nft7b" event={"ID":"9f742e7c-d921-4eff-b938-d73d798b7ec8","Type":"ContainerStarted","Data":"063d15a7c65ebe0657688ef0298d70dd1122ab3c3ff0771e8e5a40b4665bc673"}
Jul 13 19:19:36.168695 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:19:36.168607    2565 generic.go:356] "Generic (PLEG): container finished" podID="9f742e7c-d921-4eff-b938-d73d798b7ec8" containerID="f5dc0f57f4a882fba1d322373cb51ca6970ca28b6b75a3c3abf6fdea29d6ecc6" exitCode=0
Jul 13 19:19:36.169091 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:19:36.168687    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-965ccd984-nft7b" event={"ID":"9f742e7c-d921-4eff-b938-d73d798b7ec8","Type":"ContainerDied","Data":"f5dc0f57f4a882fba1d322373cb51ca6970ca28b6b75a3c3abf6fdea29d6ecc6"}
Jul 13 19:19:37.176619 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:19:37.176580    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-965ccd984-nft7b" event={"ID":"9f742e7c-d921-4eff-b938-d73d798b7ec8","Type":"ContainerStarted","Data":"650bb4ed159e3b7000535911303b8768d46d6bb59dfc6e3ae186190091c88f8d"}
Jul 13 19:19:37.202029 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:19:37.201968    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-965ccd984-nft7b" podStartSLOduration=2.502247789 podStartE2EDuration="6.201945817s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.699698028s" isStatefulPod=false podCreationTimestamp="2026-07-13 19:19:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 19:19:37.199179125 +0000 UTC m=+1763.724312562" watchObservedRunningTime="2026-07-13 19:19:37.201945817 +0000 UTC m=+1763.727079234"
Jul 13 19:19:37.931528 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:19:37.931491    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-965ccd984-nft7b"
Jul 13 19:19:41.931128 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:19:41.931089    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-965ccd984-nft7b"
Jul 13 19:19:41.932970 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:19:41.932940    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-965ccd984-nft7b" podUID="9f742e7c-d921-4eff-b938-d73d798b7ec8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": dial tcp 10.134.0.51:8000: connect: connection refused"
Jul 13 19:19:51.932153 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:19:51.932105    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-965ccd984-nft7b" podUID="9f742e7c-d921-4eff-b938-d73d798b7ec8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": dial tcp 10.134.0.51:8000: connect: connection refused"
Jul 13 19:19:54.808576 ip-10-0-128-215 kubenswrapper[2565]: E0713 19:19:54.808519    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dmww7f" podUID="634b78c4-751c-4880-9f6c-1ea69b737987"
Jul 13 19:19:55.275395 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:19:55.275360    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dmww7f"
Jul 13 19:20:01.932154 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:20:01.932107    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-965ccd984-nft7b" podUID="9f742e7c-d921-4eff-b938-d73d798b7ec8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": dial tcp 10.134.0.51:8000: connect: connection refused"
Jul 13 19:20:11.931739 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:20:11.931675    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-965ccd984-nft7b" podUID="9f742e7c-d921-4eff-b938-d73d798b7ec8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": dial tcp 10.134.0.51:8000: connect: connection refused"
Jul 13 19:20:14.031620 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:20:14.031591    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2bscc_2f0655bd-4a09-4314-8c1b-e30191eaec0e/ovn-acl-logging/0.log"
Jul 13 19:20:14.032871 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:20:14.032845    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2bscc_2f0655bd-4a09-4314-8c1b-e30191eaec0e/ovn-acl-logging/0.log"
Jul 13 19:20:21.931489 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:20:21.931397    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-965ccd984-nft7b" podUID="9f742e7c-d921-4eff-b938-d73d798b7ec8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": dial tcp 10.134.0.51:8000: connect: connection refused"
Jul 13 19:20:31.932180 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:20:31.932089    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-965ccd984-nft7b" podUID="9f742e7c-d921-4eff-b938-d73d798b7ec8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": dial tcp 10.134.0.51:8000: connect: connection refused"
Jul 13 19:20:41.931788 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:20:41.931740    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-965ccd984-nft7b" podUID="9f742e7c-d921-4eff-b938-d73d798b7ec8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": dial tcp 10.134.0.51:8000: connect: connection refused"
Jul 13 19:20:51.931784 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:20:51.931744    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-965ccd984-nft7b" podUID="9f742e7c-d921-4eff-b938-d73d798b7ec8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": dial tcp 10.134.0.51:8000: connect: connection refused"
Jul 13 19:21:01.931495 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:21:01.931451    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-965ccd984-nft7b" podUID="9f742e7c-d921-4eff-b938-d73d798b7ec8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": dial tcp 10.134.0.51:8000: connect: connection refused"
Jul 13 19:21:11.931621 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:21:11.931574    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-965ccd984-nft7b" podUID="9f742e7c-d921-4eff-b938-d73d798b7ec8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": dial tcp 10.134.0.51:8000: connect: connection refused"
Jul 13 19:21:21.931446 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:21:21.931391    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-965ccd984-nft7b" podUID="9f742e7c-d921-4eff-b938-d73d798b7ec8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": dial tcp 10.134.0.51:8000: connect: connection refused"
Jul 13 19:21:31.942020 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:21:31.941987    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-965ccd984-nft7b"
Jul 13 19:21:31.950464 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:21:31.950428    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-965ccd984-nft7b"
Jul 13 19:21:45.258196 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:21:45.258166    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-965ccd984-nft7b"]
Jul 13 19:21:58.275927 ip-10-0-128-215 kubenswrapper[2565]: E0713 19:21:58.275890    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dmww7f" podUID="634b78c4-751c-4880-9f6c-1ea69b737987"
Jul 13 19:21:58.919366 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:21:58.919336    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dmww7f"
Jul 13 19:22:00.281337 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:22:00.281294    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-965ccd984-nft7b" podUID="9f742e7c-d921-4eff-b938-d73d798b7ec8" containerName="main" containerID="cri-o://650bb4ed159e3b7000535911303b8768d46d6bb59dfc6e3ae186190091c88f8d" gracePeriod=45
Jul 13 19:22:00.930855 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:22:00.930823    2565 generic.go:356] "Generic (PLEG): container finished" podID="9f742e7c-d921-4eff-b938-d73d798b7ec8" containerID="650bb4ed159e3b7000535911303b8768d46d6bb59dfc6e3ae186190091c88f8d" exitCode=0
Jul 13 19:22:00.930989 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:22:00.930892    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-965ccd984-nft7b" event={"ID":"9f742e7c-d921-4eff-b938-d73d798b7ec8","Type":"ContainerDied","Data":"650bb4ed159e3b7000535911303b8768d46d6bb59dfc6e3ae186190091c88f8d"}
Jul 13 19:22:00.930989 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:22:00.930955    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-965ccd984-nft7b" event={"ID":"9f742e7c-d921-4eff-b938-d73d798b7ec8","Type":"ContainerDied","Data":"063d15a7c65ebe0657688ef0298d70dd1122ab3c3ff0771e8e5a40b4665bc673"}
Jul 13 19:22:00.930989 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:22:00.930977    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="063d15a7c65ebe0657688ef0298d70dd1122ab3c3ff0771e8e5a40b4665bc673"
Jul 13 19:22:00.939947 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:22:00.939928    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-965ccd984-nft7b"
Jul 13 19:22:01.042232 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:22:01.042162    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f742e7c-d921-4eff-b938-d73d798b7ec8-model-cache\") pod \"9f742e7c-d921-4eff-b938-d73d798b7ec8\" (UID: \"9f742e7c-d921-4eff-b938-d73d798b7ec8\") "
Jul 13 19:22:01.042232 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:22:01.042198    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9f742e7c-d921-4eff-b938-d73d798b7ec8-kserve-provision-location\") pod \"9f742e7c-d921-4eff-b938-d73d798b7ec8\" (UID: \"9f742e7c-d921-4eff-b938-d73d798b7ec8\") "
Jul 13 19:22:01.042434 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:22:01.042250    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f742e7c-d921-4eff-b938-d73d798b7ec8-dshm\") pod \"9f742e7c-d921-4eff-b938-d73d798b7ec8\" (UID: \"9f742e7c-d921-4eff-b938-d73d798b7ec8\") "
Jul 13 19:22:01.042434 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:22:01.042277    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gpp2q\" (UniqueName: \"kubernetes.io/projected/9f742e7c-d921-4eff-b938-d73d798b7ec8-kube-api-access-gpp2q\") pod \"9f742e7c-d921-4eff-b938-d73d798b7ec8\" (UID: \"9f742e7c-d921-4eff-b938-d73d798b7ec8\") "
Jul 13 19:22:01.042434 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:22:01.042293    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f742e7c-d921-4eff-b938-d73d798b7ec8-tls-certs\") pod \"9f742e7c-d921-4eff-b938-d73d798b7ec8\" (UID: \"9f742e7c-d921-4eff-b938-d73d798b7ec8\") "
Jul 13 19:22:01.042434 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:22:01.042316    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f742e7c-d921-4eff-b938-d73d798b7ec8-home\") pod \"9f742e7c-d921-4eff-b938-d73d798b7ec8\" (UID: \"9f742e7c-d921-4eff-b938-d73d798b7ec8\") "
Jul 13 19:22:01.042652 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:22:01.042442    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f742e7c-d921-4eff-b938-d73d798b7ec8-model-cache" (OuterVolumeSpecName: "model-cache") pod "9f742e7c-d921-4eff-b938-d73d798b7ec8" (UID: "9f742e7c-d921-4eff-b938-d73d798b7ec8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:22:01.042652 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:22:01.042463    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f742e7c-d921-4eff-b938-d73d798b7ec8-tmp-dir\") pod \"9f742e7c-d921-4eff-b938-d73d798b7ec8\" (UID: \"9f742e7c-d921-4eff-b938-d73d798b7ec8\") "
Jul 13 19:22:01.042821 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:22:01.042790    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f742e7c-d921-4eff-b938-d73d798b7ec8-model-cache\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 13 19:22:01.043226 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:22:01.043201    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f742e7c-d921-4eff-b938-d73d798b7ec8-home" (OuterVolumeSpecName: "home") pod "9f742e7c-d921-4eff-b938-d73d798b7ec8" (UID: "9f742e7c-d921-4eff-b938-d73d798b7ec8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:22:01.044607 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:22:01.044577    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f742e7c-d921-4eff-b938-d73d798b7ec8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9f742e7c-d921-4eff-b938-d73d798b7ec8" (UID: "9f742e7c-d921-4eff-b938-d73d798b7ec8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 19:22:01.044607 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:22:01.044581    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f742e7c-d921-4eff-b938-d73d798b7ec8-kube-api-access-gpp2q" (OuterVolumeSpecName: "kube-api-access-gpp2q") pod "9f742e7c-d921-4eff-b938-d73d798b7ec8" (UID: "9f742e7c-d921-4eff-b938-d73d798b7ec8"). InnerVolumeSpecName "kube-api-access-gpp2q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 19:22:01.044831 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:22:01.044685    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f742e7c-d921-4eff-b938-d73d798b7ec8-dshm" (OuterVolumeSpecName: "dshm") pod "9f742e7c-d921-4eff-b938-d73d798b7ec8" (UID: "9f742e7c-d921-4eff-b938-d73d798b7ec8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:22:01.054217 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:22:01.054189    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f742e7c-d921-4eff-b938-d73d798b7ec8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9f742e7c-d921-4eff-b938-d73d798b7ec8" (UID: "9f742e7c-d921-4eff-b938-d73d798b7ec8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:22:01.106802 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:22:01.106776    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f742e7c-d921-4eff-b938-d73d798b7ec8-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "9f742e7c-d921-4eff-b938-d73d798b7ec8" (UID: "9f742e7c-d921-4eff-b938-d73d798b7ec8"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:22:01.144196 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:22:01.144173    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f742e7c-d921-4eff-b938-d73d798b7ec8-home\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 13 19:22:01.144260 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:22:01.144201    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f742e7c-d921-4eff-b938-d73d798b7ec8-tmp-dir\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 13 19:22:01.144260 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:22:01.144219    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9f742e7c-d921-4eff-b938-d73d798b7ec8-kserve-provision-location\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 13 19:22:01.144260 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:22:01.144233    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f742e7c-d921-4eff-b938-d73d798b7ec8-dshm\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 13 19:22:01.144260 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:22:01.144249    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gpp2q\" (UniqueName: \"kubernetes.io/projected/9f742e7c-d921-4eff-b938-d73d798b7ec8-kube-api-access-gpp2q\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 13 19:22:01.144389 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:22:01.144269    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f742e7c-d921-4eff-b938-d73d798b7ec8-tls-certs\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 13 19:22:01.935877 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:22:01.935837    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-965ccd984-nft7b"
Jul 13 19:22:01.958924 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:22:01.958896    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-965ccd984-nft7b"]
Jul 13 19:22:01.963512 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:22:01.963486    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-965ccd984-nft7b"]
Jul 13 19:22:02.047994 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:22:02.047961    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9f742e7c-d921-4eff-b938-d73d798b7ec8" path="/var/lib/kubelet/pods/9f742e7c-d921-4eff-b938-d73d798b7ec8/volumes"
Jul 13 19:22:25.700446 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:22:25.700400    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-555479964d-4kccs"]
Jul 13 19:22:25.701021 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:22:25.701001    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f742e7c-d921-4eff-b938-d73d798b7ec8" containerName="storage-initializer"
Jul 13 19:22:25.701106 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:22:25.701025    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f742e7c-d921-4eff-b938-d73d798b7ec8" containerName="storage-initializer"
Jul 13 19:22:25.701106 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:22:25.701042    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f742e7c-d921-4eff-b938-d73d798b7ec8" containerName="main"
Jul 13 19:22:25.701106 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:22:25.701051    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f742e7c-d921-4eff-b938-d73d798b7ec8" containerName="main"
Jul 13 19:22:25.701262 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:22:25.701165    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f742e7c-d921-4eff-b938-d73d798b7ec8" containerName="main"
Jul 13 19:22:25.704231 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:22:25.704211    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-555479964d-4kccs"
Jul 13 19:22:25.706878 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:22:25.706859    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 13 19:22:25.706981 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:22:25.706936    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-with-refs-test-kserve-self-signed-certs\""
Jul 13 19:22:25.715037 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:22:25.715015    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-555479964d-4kccs"]
Jul 13 19:22:25.848508 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:22:25.848480    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3efdf9e-cef9-42ff-ab32-220cd931ff42-tmp-dir\") pod \"router-with-refs-test-kserve-555479964d-4kccs\" (UID: \"b3efdf9e-cef9-42ff-ab32-220cd931ff42\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-555479964d-4kccs"
Jul 13 19:22:25.848688 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:22:25.848514    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b3efdf9e-cef9-42ff-ab32-220cd931ff42-tls-certs\") pod \"router-with-refs-test-kserve-555479964d-4kccs\" (UID: \"b3efdf9e-cef9-42ff-ab32-220cd931ff42\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-555479964d-4kccs"
Jul 13 19:22:25.848688 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:22:25.848574    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b3efdf9e-cef9-42ff-ab32-220cd931ff42-model-cache\") pod \"router-with-refs-test-kserve-555479964d-4kccs\" (UID: \"b3efdf9e-cef9-42ff-ab32-220cd931ff42\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-555479964d-4kccs"
Jul 13 19:22:25.848688 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:22:25.848605    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b3efdf9e-cef9-42ff-ab32-220cd931ff42-dshm\") pod \"router-with-refs-test-kserve-555479964d-4kccs\" (UID: \"b3efdf9e-cef9-42ff-ab32-220cd931ff42\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-555479964d-4kccs"
Jul 13 19:22:25.848688 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:22:25.848629    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b3efdf9e-cef9-42ff-ab32-220cd931ff42-kserve-provision-location\") pod \"router-with-refs-test-kserve-555479964d-4kccs\" (UID: \"b3efdf9e-cef9-42ff-ab32-220cd931ff42\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-555479964d-4kccs"
Jul 13 19:22:25.848688 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:22:25.848656    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b3efdf9e-cef9-42ff-ab32-220cd931ff42-home\") pod \"router-with-refs-test-kserve-555479964d-4kccs\" (UID: \"b3efdf9e-cef9-42ff-ab32-220cd931ff42\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-555479964d-4kccs"
Jul 13 19:22:25.848913 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:22:25.848760    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chwbw\" (UniqueName: \"kubernetes.io/projected/b3efdf9e-cef9-42ff-ab32-220cd931ff42-kube-api-access-chwbw\") pod \"router-with-refs-test-kserve-555479964d-4kccs\" (UID: \"b3efdf9e-cef9-42ff-ab32-220cd931ff42\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-555479964d-4kccs"
Jul 13 19:22:25.949905 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:22:25.949874    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3efdf9e-cef9-42ff-ab32-220cd931ff42-tmp-dir\") pod \"router-with-refs-test-kserve-555479964d-4kccs\" (UID: \"b3efdf9e-cef9-42ff-ab32-220cd931ff42\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-555479964d-4kccs"
Jul 13 19:22:25.949905 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:22:25.949903    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b3efdf9e-cef9-42ff-ab32-220cd931ff42-tls-certs\") pod \"router-with-refs-test-kserve-555479964d-4kccs\" (UID: \"b3efdf9e-cef9-42ff-ab32-220cd931ff42\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-555479964d-4kccs"
Jul 13 19:22:25.950105 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:22:25.949940    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b3efdf9e-cef9-42ff-ab32-220cd931ff42-model-cache\") pod \"router-with-refs-test-kserve-555479964d-4kccs\" (UID: \"b3efdf9e-cef9-42ff-ab32-220cd931ff42\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-555479964d-4kccs"
Jul 13 19:22:25.950105 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:22:25.949977    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b3efdf9e-cef9-42ff-ab32-220cd931ff42-dshm\") pod \"router-with-refs-test-kserve-555479964d-4kccs\" (UID: \"b3efdf9e-cef9-42ff-ab32-220cd931ff42\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-555479964d-4kccs"
Jul 13 19:22:25.950105 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:22:25.950001    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b3efdf9e-cef9-42ff-ab32-220cd931ff42-kserve-provision-location\") pod \"router-with-refs-test-kserve-555479964d-4kccs\" (UID: \"b3efdf9e-cef9-42ff-ab32-220cd931ff42\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-555479964d-4kccs"
Jul 13 19:22:25.950105 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:22:25.950027    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b3efdf9e-cef9-42ff-ab32-220cd931ff42-home\") pod \"router-with-refs-test-kserve-555479964d-4kccs\" (UID: \"b3efdf9e-cef9-42ff-ab32-220cd931ff42\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-555479964d-4kccs"
Jul 13 19:22:25.950105 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:22:25.950071    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chwbw\" (UniqueName: \"kubernetes.io/projected/b3efdf9e-cef9-42ff-ab32-220cd931ff42-kube-api-access-chwbw\") pod \"router-with-refs-test-kserve-555479964d-4kccs\" (UID: \"b3efdf9e-cef9-42ff-ab32-220cd931ff42\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-555479964d-4kccs"
Jul 13 19:22:25.950409 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:22:25.950302    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3efdf9e-cef9-42ff-ab32-220cd931ff42-tmp-dir\") pod \"router-with-refs-test-kserve-555479964d-4kccs\" (UID: \"b3efdf9e-cef9-42ff-ab32-220cd931ff42\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-555479964d-4kccs"
Jul 13 19:22:25.950409 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:22:25.950317    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b3efdf9e-cef9-42ff-ab32-220cd931ff42-model-cache\") pod \"router-with-refs-test-kserve-555479964d-4kccs\" (UID: \"b3efdf9e-cef9-42ff-ab32-220cd931ff42\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-555479964d-4kccs"
Jul 13 19:22:25.950553 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:22:25.950480    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b3efdf9e-cef9-42ff-ab32-220cd931ff42-kserve-provision-location\") pod \"router-with-refs-test-kserve-555479964d-4kccs\" (UID: \"b3efdf9e-cef9-42ff-ab32-220cd931ff42\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-555479964d-4kccs"
Jul 13 19:22:25.950553 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:22:25.950487    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b3efdf9e-cef9-42ff-ab32-220cd931ff42-home\") pod \"router-with-refs-test-kserve-555479964d-4kccs\" (UID: \"b3efdf9e-cef9-42ff-ab32-220cd931ff42\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-555479964d-4kccs"
Jul 13 19:22:25.952449 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:22:25.952429    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b3efdf9e-cef9-42ff-ab32-220cd931ff42-dshm\") pod \"router-with-refs-test-kserve-555479964d-4kccs\" (UID: \"b3efdf9e-cef9-42ff-ab32-220cd931ff42\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-555479964d-4kccs"
Jul 13 19:22:25.952558 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:22:25.952533    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b3efdf9e-cef9-42ff-ab32-220cd931ff42-tls-certs\") pod \"router-with-refs-test-kserve-555479964d-4kccs\" (UID: \"b3efdf9e-cef9-42ff-ab32-220cd931ff42\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-555479964d-4kccs"
Jul 13 19:22:25.960726 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:22:25.960686    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-chwbw\" (UniqueName: \"kubernetes.io/projected/b3efdf9e-cef9-42ff-ab32-220cd931ff42-kube-api-access-chwbw\") pod \"router-with-refs-test-kserve-555479964d-4kccs\" (UID: \"b3efdf9e-cef9-42ff-ab32-220cd931ff42\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-555479964d-4kccs"
Jul 13 19:22:26.015487 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:22:26.015463    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-555479964d-4kccs"
Jul 13 19:22:26.153934 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:22:26.153906    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-555479964d-4kccs"]
Jul 13 19:22:26.155354 ip-10-0-128-215 kubenswrapper[2565]: W0713 19:22:26.155331    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3efdf9e_cef9_42ff_ab32_220cd931ff42.slice/crio-d45e9b5326c36e07a7c0cdf4a24125886e0394b76fccc2dfe71076e58a81aaff WatchSource:0}: Error finding container d45e9b5326c36e07a7c0cdf4a24125886e0394b76fccc2dfe71076e58a81aaff: Status 404 returned error can't find the container with id d45e9b5326c36e07a7c0cdf4a24125886e0394b76fccc2dfe71076e58a81aaff
Jul 13 19:22:27.056041 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:22:27.055999    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-555479964d-4kccs" event={"ID":"b3efdf9e-cef9-42ff-ab32-220cd931ff42","Type":"ContainerStarted","Data":"799b6909a53143b2bb6884d49888e16d0875e971976c3ff4c13659e2471a246e"}
Jul 13 19:22:27.056524 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:22:27.056075    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-555479964d-4kccs" event={"ID":"b3efdf9e-cef9-42ff-ab32-220cd931ff42","Type":"ContainerStarted","Data":"d45e9b5326c36e07a7c0cdf4a24125886e0394b76fccc2dfe71076e58a81aaff"}
Jul 13 19:22:30.071217 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:22:30.071184    2565 generic.go:356] "Generic (PLEG): container finished" podID="b3efdf9e-cef9-42ff-ab32-220cd931ff42" containerID="799b6909a53143b2bb6884d49888e16d0875e971976c3ff4c13659e2471a246e" exitCode=0
Jul 13 19:22:30.071597 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:22:30.071251    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-555479964d-4kccs" event={"ID":"b3efdf9e-cef9-42ff-ab32-220cd931ff42","Type":"ContainerDied","Data":"799b6909a53143b2bb6884d49888e16d0875e971976c3ff4c13659e2471a246e"}
Jul 13 19:22:31.079323 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:22:31.079285    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-555479964d-4kccs" event={"ID":"b3efdf9e-cef9-42ff-ab32-220cd931ff42","Type":"ContainerStarted","Data":"06ccb16042b93ce06473c51825e3ab56ef662c285410f1d96ff612ba9af0f430"}
Jul 13 19:22:31.103690 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:22:31.103635    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-555479964d-4kccs" podStartSLOduration=3.227712789 podStartE2EDuration="6.103620494s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.875907705s" isStatefulPod=false podCreationTimestamp="2026-07-13 19:22:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 19:22:31.101820766 +0000 UTC m=+1937.626954195" watchObservedRunningTime="2026-07-13 19:22:31.103620494 +0000 UTC m=+1937.628753908"
Jul 13 19:22:32.015718 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:22:32.015671    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-555479964d-4kccs"
Jul 13 19:22:36.016483 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:22:36.016448    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-555479964d-4kccs"
Jul 13 19:22:36.017876 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:22:36.017849    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-555479964d-4kccs" podUID="b3efdf9e-cef9-42ff-ab32-220cd931ff42" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": dial tcp 10.134.0.52:8000: connect: connection refused"
Jul 13 19:22:46.016725 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:22:46.016654    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-555479964d-4kccs" podUID="b3efdf9e-cef9-42ff-ab32-220cd931ff42" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": dial tcp 10.134.0.52:8000: connect: connection refused"
Jul 13 19:22:56.016231 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:22:56.016179    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-555479964d-4kccs" podUID="b3efdf9e-cef9-42ff-ab32-220cd931ff42" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": dial tcp 10.134.0.52:8000: connect: connection refused"
Jul 13 19:23:06.017809 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:23:06.016751    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-555479964d-4kccs" podUID="b3efdf9e-cef9-42ff-ab32-220cd931ff42" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": dial tcp 10.134.0.52:8000: connect: connection refused"
Jul 13 19:23:16.016797 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:23:16.016755    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-555479964d-4kccs" podUID="b3efdf9e-cef9-42ff-ab32-220cd931ff42" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": dial tcp 10.134.0.52:8000: connect: connection refused"
Jul 13 19:23:26.016114 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:23:26.016055    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-555479964d-4kccs" podUID="b3efdf9e-cef9-42ff-ab32-220cd931ff42" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": dial tcp 10.134.0.52:8000: connect: connection refused"
Jul 13 19:23:36.016539 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:23:36.016496    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-555479964d-4kccs" podUID="b3efdf9e-cef9-42ff-ab32-220cd931ff42" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": dial tcp 10.134.0.52:8000: connect: connection refused"
Jul 13 19:23:46.015945 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:23:46.015889    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-555479964d-4kccs" podUID="b3efdf9e-cef9-42ff-ab32-220cd931ff42" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": dial tcp 10.134.0.52:8000: connect: connection refused"
Jul 13 19:23:56.016595 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:23:56.016541    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-555479964d-4kccs" podUID="b3efdf9e-cef9-42ff-ab32-220cd931ff42" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": dial tcp 10.134.0.52:8000: connect: connection refused"
Jul 13 19:24:01.920724 ip-10-0-128-215 kubenswrapper[2565]: E0713 19:24:01.920671    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dmww7f" podUID="634b78c4-751c-4880-9f6c-1ea69b737987"
Jul 13 19:24:02.557542 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:24:02.557504    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dmww7f"
Jul 13 19:24:06.016221 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:24:06.016179    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-555479964d-4kccs" podUID="b3efdf9e-cef9-42ff-ab32-220cd931ff42" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": dial tcp 10.134.0.52:8000: connect: connection refused"
Jul 13 19:24:16.016821 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:24:16.016764    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-555479964d-4kccs" podUID="b3efdf9e-cef9-42ff-ab32-220cd931ff42" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": dial tcp 10.134.0.52:8000: connect: connection refused"
Jul 13 19:24:26.026273 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:24:26.026245    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-555479964d-4kccs"
Jul 13 19:24:26.034204 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:24:26.034182    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-555479964d-4kccs"
Jul 13 19:25:14.064494 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:25:14.064466    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2bscc_2f0655bd-4a09-4314-8c1b-e30191eaec0e/ovn-acl-logging/0.log"
Jul 13 19:25:14.067886 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:25:14.067863    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2bscc_2f0655bd-4a09-4314-8c1b-e30191eaec0e/ovn-acl-logging/0.log"
Jul 13 19:25:31.465122 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:25:31.465078    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-555479964d-4kccs"]
Jul 13 19:25:46.490251 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:25:46.490185    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-555479964d-4kccs" podUID="b3efdf9e-cef9-42ff-ab32-220cd931ff42" containerName="main" containerID="cri-o://06ccb16042b93ce06473c51825e3ab56ef662c285410f1d96ff612ba9af0f430" gracePeriod=45
Jul 13 19:25:47.027649 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:25:47.027614    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-555479964d-4kccs" podUID="b3efdf9e-cef9-42ff-ab32-220cd931ff42" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": dial tcp 10.134.0.52:8000: connect: connection refused"
Jul 13 19:25:47.248899 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:25:47.248839    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-555479964d-4kccs"
Jul 13 19:25:47.368610 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:25:47.368584    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b3efdf9e-cef9-42ff-ab32-220cd931ff42-dshm\") pod \"b3efdf9e-cef9-42ff-ab32-220cd931ff42\" (UID: \"b3efdf9e-cef9-42ff-ab32-220cd931ff42\") "
Jul 13 19:25:47.368814 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:25:47.368628    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b3efdf9e-cef9-42ff-ab32-220cd931ff42-tls-certs\") pod \"b3efdf9e-cef9-42ff-ab32-220cd931ff42\" (UID: \"b3efdf9e-cef9-42ff-ab32-220cd931ff42\") "
Jul 13 19:25:47.368814 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:25:47.368668    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b3efdf9e-cef9-42ff-ab32-220cd931ff42-kserve-provision-location\") pod \"b3efdf9e-cef9-42ff-ab32-220cd931ff42\" (UID: \"b3efdf9e-cef9-42ff-ab32-220cd931ff42\") "
Jul 13 19:25:47.368814 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:25:47.368730    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b3efdf9e-cef9-42ff-ab32-220cd931ff42-model-cache\") pod \"b3efdf9e-cef9-42ff-ab32-220cd931ff42\" (UID: \"b3efdf9e-cef9-42ff-ab32-220cd931ff42\") "
Jul 13 19:25:47.368814 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:25:47.368758    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b3efdf9e-cef9-42ff-ab32-220cd931ff42-home\") pod \"b3efdf9e-cef9-42ff-ab32-220cd931ff42\" (UID: \"b3efdf9e-cef9-42ff-ab32-220cd931ff42\") "
Jul 13 19:25:47.368987 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:25:47.368829    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-chwbw\" (UniqueName: \"kubernetes.io/projected/b3efdf9e-cef9-42ff-ab32-220cd931ff42-kube-api-access-chwbw\") pod \"b3efdf9e-cef9-42ff-ab32-220cd931ff42\" (UID: \"b3efdf9e-cef9-42ff-ab32-220cd931ff42\") "
Jul 13 19:25:47.368987 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:25:47.368870    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3efdf9e-cef9-42ff-ab32-220cd931ff42-tmp-dir\") pod \"b3efdf9e-cef9-42ff-ab32-220cd931ff42\" (UID: \"b3efdf9e-cef9-42ff-ab32-220cd931ff42\") "
Jul 13 19:25:47.369062 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:25:47.369004    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3efdf9e-cef9-42ff-ab32-220cd931ff42-model-cache" (OuterVolumeSpecName: "model-cache") pod "b3efdf9e-cef9-42ff-ab32-220cd931ff42" (UID: "b3efdf9e-cef9-42ff-ab32-220cd931ff42"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:25:47.369241 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:25:47.369211    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b3efdf9e-cef9-42ff-ab32-220cd931ff42-model-cache\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 13 19:25:47.369458 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:25:47.369437    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3efdf9e-cef9-42ff-ab32-220cd931ff42-home" (OuterVolumeSpecName: "home") pod "b3efdf9e-cef9-42ff-ab32-220cd931ff42" (UID: "b3efdf9e-cef9-42ff-ab32-220cd931ff42"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:25:47.371130 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:25:47.371107    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3efdf9e-cef9-42ff-ab32-220cd931ff42-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b3efdf9e-cef9-42ff-ab32-220cd931ff42" (UID: "b3efdf9e-cef9-42ff-ab32-220cd931ff42"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 19:25:47.371520 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:25:47.371488    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3efdf9e-cef9-42ff-ab32-220cd931ff42-kube-api-access-chwbw" (OuterVolumeSpecName: "kube-api-access-chwbw") pod "b3efdf9e-cef9-42ff-ab32-220cd931ff42" (UID: "b3efdf9e-cef9-42ff-ab32-220cd931ff42"). InnerVolumeSpecName "kube-api-access-chwbw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 19:25:47.371520 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:25:47.371487    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3efdf9e-cef9-42ff-ab32-220cd931ff42-dshm" (OuterVolumeSpecName: "dshm") pod "b3efdf9e-cef9-42ff-ab32-220cd931ff42" (UID: "b3efdf9e-cef9-42ff-ab32-220cd931ff42"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:25:47.382579 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:25:47.382550    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3efdf9e-cef9-42ff-ab32-220cd931ff42-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b3efdf9e-cef9-42ff-ab32-220cd931ff42" (UID: "b3efdf9e-cef9-42ff-ab32-220cd931ff42"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:25:47.435554 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:25:47.435517    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3efdf9e-cef9-42ff-ab32-220cd931ff42-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b3efdf9e-cef9-42ff-ab32-220cd931ff42" (UID: "b3efdf9e-cef9-42ff-ab32-220cd931ff42"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:25:47.469671 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:25:47.469642    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b3efdf9e-cef9-42ff-ab32-220cd931ff42-tls-certs\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 13 19:25:47.469671 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:25:47.469667    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b3efdf9e-cef9-42ff-ab32-220cd931ff42-kserve-provision-location\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 13 19:25:47.469831 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:25:47.469682    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b3efdf9e-cef9-42ff-ab32-220cd931ff42-home\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 13 19:25:47.469831 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:25:47.469695    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-chwbw\" (UniqueName: \"kubernetes.io/projected/b3efdf9e-cef9-42ff-ab32-220cd931ff42-kube-api-access-chwbw\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 13 19:25:47.469831 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:25:47.469724    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3efdf9e-cef9-42ff-ab32-220cd931ff42-tmp-dir\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 13 19:25:47.469831 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:25:47.469733    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b3efdf9e-cef9-42ff-ab32-220cd931ff42-dshm\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 13 19:25:48.089946 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:25:48.089916    2565 generic.go:356] "Generic (PLEG): container finished" podID="b3efdf9e-cef9-42ff-ab32-220cd931ff42" containerID="06ccb16042b93ce06473c51825e3ab56ef662c285410f1d96ff612ba9af0f430" exitCode=0
Jul 13 19:25:48.090332 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:25:48.089966    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-555479964d-4kccs"
Jul 13 19:25:48.090332 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:25:48.089978    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-555479964d-4kccs" event={"ID":"b3efdf9e-cef9-42ff-ab32-220cd931ff42","Type":"ContainerDied","Data":"06ccb16042b93ce06473c51825e3ab56ef662c285410f1d96ff612ba9af0f430"}
Jul 13 19:25:48.090332 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:25:48.090023    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-555479964d-4kccs" event={"ID":"b3efdf9e-cef9-42ff-ab32-220cd931ff42","Type":"ContainerDied","Data":"d45e9b5326c36e07a7c0cdf4a24125886e0394b76fccc2dfe71076e58a81aaff"}
Jul 13 19:25:48.090332 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:25:48.090048    2565 scope.go:117] "RemoveContainer" containerID="06ccb16042b93ce06473c51825e3ab56ef662c285410f1d96ff612ba9af0f430"
Jul 13 19:25:48.098928 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:25:48.098905    2565 scope.go:117] "RemoveContainer" containerID="799b6909a53143b2bb6884d49888e16d0875e971976c3ff4c13659e2471a246e"
Jul 13 19:25:48.107044 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:25:48.107027    2565 scope.go:117] "RemoveContainer" containerID="06ccb16042b93ce06473c51825e3ab56ef662c285410f1d96ff612ba9af0f430"
Jul 13 19:25:48.107317 ip-10-0-128-215 kubenswrapper[2565]: E0713 19:25:48.107294    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06ccb16042b93ce06473c51825e3ab56ef662c285410f1d96ff612ba9af0f430\": container with ID starting with 06ccb16042b93ce06473c51825e3ab56ef662c285410f1d96ff612ba9af0f430 not found: ID does not exist" containerID="06ccb16042b93ce06473c51825e3ab56ef662c285410f1d96ff612ba9af0f430"
Jul 13 19:25:48.107420 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:25:48.107323    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06ccb16042b93ce06473c51825e3ab56ef662c285410f1d96ff612ba9af0f430"} err="failed to get container status \"06ccb16042b93ce06473c51825e3ab56ef662c285410f1d96ff612ba9af0f430\": rpc error: code = NotFound desc = could not find container \"06ccb16042b93ce06473c51825e3ab56ef662c285410f1d96ff612ba9af0f430\": container with ID starting with 06ccb16042b93ce06473c51825e3ab56ef662c285410f1d96ff612ba9af0f430 not found: ID does not exist"
Jul 13 19:25:48.107420 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:25:48.107339    2565 scope.go:117] "RemoveContainer" containerID="799b6909a53143b2bb6884d49888e16d0875e971976c3ff4c13659e2471a246e"
Jul 13 19:25:48.107589 ip-10-0-128-215 kubenswrapper[2565]: E0713 19:25:48.107568    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"799b6909a53143b2bb6884d49888e16d0875e971976c3ff4c13659e2471a246e\": container with ID starting with 799b6909a53143b2bb6884d49888e16d0875e971976c3ff4c13659e2471a246e not found: ID does not exist" containerID="799b6909a53143b2bb6884d49888e16d0875e971976c3ff4c13659e2471a246e"
Jul 13 19:25:48.107691 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:25:48.107596    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"799b6909a53143b2bb6884d49888e16d0875e971976c3ff4c13659e2471a246e"} err="failed to get container status \"799b6909a53143b2bb6884d49888e16d0875e971976c3ff4c13659e2471a246e\": rpc error: code = NotFound desc = could not find container \"799b6909a53143b2bb6884d49888e16d0875e971976c3ff4c13659e2471a246e\": container with ID starting with 799b6909a53143b2bb6884d49888e16d0875e971976c3ff4c13659e2471a246e not found: ID does not exist"
Jul 13 19:25:48.110548 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:25:48.110524    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-555479964d-4kccs"]
Jul 13 19:25:48.114313 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:25:48.114295    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-555479964d-4kccs"]
Jul 13 19:25:50.048852 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:25:50.048818    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b3efdf9e-cef9-42ff-ab32-220cd931ff42" path="/var/lib/kubelet/pods/b3efdf9e-cef9-42ff-ab32-220cd931ff42/volumes"
Jul 13 19:26:02.370095 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:26:02.370056    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-84599bcd6b4vvr6"]
Jul 13 19:26:02.370521 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:26:02.370505    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b3efdf9e-cef9-42ff-ab32-220cd931ff42" containerName="main"
Jul 13 19:26:02.370574 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:26:02.370524    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="b3efdf9e-cef9-42ff-ab32-220cd931ff42" containerName="main"
Jul 13 19:26:02.370574 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:26:02.370540    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b3efdf9e-cef9-42ff-ab32-220cd931ff42" containerName="storage-initializer"
Jul 13 19:26:02.370574 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:26:02.370545    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="b3efdf9e-cef9-42ff-ab32-220cd931ff42" containerName="storage-initializer"
Jul 13 19:26:02.370729 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:26:02.370657    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="b3efdf9e-cef9-42ff-ab32-220cd931ff42" containerName="main"
Jul 13 19:26:02.375828 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:26:02.375806    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-84599bcd6b4vvr6"
Jul 13 19:26:02.378545 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:26:02.378517    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv44d181485fad85e662eb092f3749502f-kserve-self-signed-certs\""
Jul 13 19:26:02.378683 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:26:02.378625    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 13 19:26:02.378683 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:26:02.378640    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-dockercfg-4gfrg\""
Jul 13 19:26:02.390868 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:26:02.390843    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-84599bcd6b4vvr6"]
Jul 13 19:26:02.398093 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:26:02.398067    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f654d9"]
Jul 13 19:26:02.401675 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:26:02.401658    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f654d9"
Jul 13 19:26:02.415918 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:26:02.415894    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f654d9"]
Jul 13 19:26:02.499240 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:26:02.499202    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5136b300-bb16-4ca4-a456-bc183d99dca9-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-84599bcd6b4vvr6\" (UID: \"5136b300-bb16-4ca4-a456-bc183d99dca9\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-84599bcd6b4vvr6"
Jul 13 19:26:02.499240 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:26:02.499239    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f8835928-0e7f-495c-9cee-96db95f344df-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f654d9\" (UID: \"f8835928-0e7f-495c-9cee-96db95f344df\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f654d9"
Jul 13 19:26:02.499470 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:26:02.499269    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mj4xt\" (UniqueName: \"kubernetes.io/projected/5136b300-bb16-4ca4-a456-bc183d99dca9-kube-api-access-mj4xt\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-84599bcd6b4vvr6\" (UID: \"5136b300-bb16-4ca4-a456-bc183d99dca9\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-84599bcd6b4vvr6"
Jul 13 19:26:02.499470 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:26:02.499358    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f8835928-0e7f-495c-9cee-96db95f344df-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f654d9\" (UID: \"f8835928-0e7f-495c-9cee-96db95f344df\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f654d9"
Jul 13 19:26:02.499470 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:26:02.499393    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5136b300-bb16-4ca4-a456-bc183d99dca9-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-84599bcd6b4vvr6\" (UID: \"5136b300-bb16-4ca4-a456-bc183d99dca9\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-84599bcd6b4vvr6"
Jul 13 19:26:02.499470 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:26:02.499412    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5qw2\" (UniqueName: \"kubernetes.io/projected/f8835928-0e7f-495c-9cee-96db95f344df-kube-api-access-z5qw2\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f654d9\" (UID: \"f8835928-0e7f-495c-9cee-96db95f344df\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f654d9"
Jul 13 19:26:02.499470 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:26:02.499431    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f8835928-0e7f-495c-9cee-96db95f344df-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f654d9\" (UID: \"f8835928-0e7f-495c-9cee-96db95f344df\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f654d9"
Jul 13 19:26:02.499470 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:26:02.499464    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5136b300-bb16-4ca4-a456-bc183d99dca9-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-84599bcd6b4vvr6\" (UID: \"5136b300-bb16-4ca4-a456-bc183d99dca9\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-84599bcd6b4vvr6"
Jul 13 19:26:02.499748 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:26:02.499490    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5136b300-bb16-4ca4-a456-bc183d99dca9-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-84599bcd6b4vvr6\" (UID: \"5136b300-bb16-4ca4-a456-bc183d99dca9\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-84599bcd6b4vvr6"
Jul 13 19:26:02.499748 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:26:02.499509    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5136b300-bb16-4ca4-a456-bc183d99dca9-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-84599bcd6b4vvr6\" (UID: \"5136b300-bb16-4ca4-a456-bc183d99dca9\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-84599bcd6b4vvr6"
Jul 13 19:26:02.499748 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:26:02.499534    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8835928-0e7f-495c-9cee-96db95f344df-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f654d9\" (UID: \"f8835928-0e7f-495c-9cee-96db95f344df\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f654d9"
Jul 13 19:26:02.499748 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:26:02.499549    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8835928-0e7f-495c-9cee-96db95f344df-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f654d9\" (UID: \"f8835928-0e7f-495c-9cee-96db95f344df\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f654d9"
Jul 13 19:26:02.499748 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:26:02.499608    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8835928-0e7f-495c-9cee-96db95f344df-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f654d9\" (UID: \"f8835928-0e7f-495c-9cee-96db95f344df\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f654d9"
Jul 13 19:26:02.499748 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:26:02.499637    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5136b300-bb16-4ca4-a456-bc183d99dca9-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-84599bcd6b4vvr6\" (UID: \"5136b300-bb16-4ca4-a456-bc183d99dca9\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-84599bcd6b4vvr6"
Jul 13 19:26:02.600697 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:26:02.600644    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5136b300-bb16-4ca4-a456-bc183d99dca9-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-84599bcd6b4vvr6\" (UID: \"5136b300-bb16-4ca4-a456-bc183d99dca9\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-84599bcd6b4vvr6"
Jul 13 19:26:02.600932 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:26:02.600735    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5136b300-bb16-4ca4-a456-bc183d99dca9-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-84599bcd6b4vvr6\" (UID: \"5136b300-bb16-4ca4-a456-bc183d99dca9\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-84599bcd6b4vvr6"
Jul 13 19:26:02.600932 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:26:02.600754    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5136b300-bb16-4ca4-a456-bc183d99dca9-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-84599bcd6b4vvr6\" (UID: \"5136b300-bb16-4ca4-a456-bc183d99dca9\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-84599bcd6b4vvr6"
Jul 13 19:26:02.600932 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:26:02.600780    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8835928-0e7f-495c-9cee-96db95f344df-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f654d9\" (UID: \"f8835928-0e7f-495c-9cee-96db95f344df\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f654d9"
Jul 13 19:26:02.600932 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:26:02.600799    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8835928-0e7f-495c-9cee-96db95f344df-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f654d9\" (UID: \"f8835928-0e7f-495c-9cee-96db95f344df\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f654d9"
Jul 13 19:26:02.600932 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:26:02.600824    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8835928-0e7f-495c-9cee-96db95f344df-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f654d9\" (UID: \"f8835928-0e7f-495c-9cee-96db95f344df\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f654d9"
Jul 13 19:26:02.600932 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:26:02.600862    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5136b300-bb16-4ca4-a456-bc183d99dca9-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-84599bcd6b4vvr6\" (UID: \"5136b300-bb16-4ca4-a456-bc183d99dca9\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-84599bcd6b4vvr6"
Jul 13 19:26:02.600932 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:26:02.600913    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5136b300-bb16-4ca4-a456-bc183d99dca9-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-84599bcd6b4vvr6\" (UID: \"5136b300-bb16-4ca4-a456-bc183d99dca9\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-84599bcd6b4vvr6"
Jul 13 19:26:02.601309 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:26:02.600939    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f8835928-0e7f-495c-9cee-96db95f344df-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f654d9\" (UID: \"f8835928-0e7f-495c-9cee-96db95f344df\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f654d9"
Jul 13 19:26:02.601309 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:26:02.600980    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mj4xt\" (UniqueName: \"kubernetes.io/projected/5136b300-bb16-4ca4-a456-bc183d99dca9-kube-api-access-mj4xt\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-84599bcd6b4vvr6\" (UID: \"5136b300-bb16-4ca4-a456-bc183d99dca9\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-84599bcd6b4vvr6"
Jul 13 19:26:02.601309 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:26:02.601014    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f8835928-0e7f-495c-9cee-96db95f344df-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f654d9\" (UID: \"f8835928-0e7f-495c-9cee-96db95f344df\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f654d9"
Jul 13 19:26:02.601309 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:26:02.601049    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5136b300-bb16-4ca4-a456-bc183d99dca9-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-84599bcd6b4vvr6\" (UID: \"5136b300-bb16-4ca4-a456-bc183d99dca9\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-84599bcd6b4vvr6"
Jul 13 19:26:02.601309 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:26:02.601078    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5qw2\" (UniqueName: \"kubernetes.io/projected/f8835928-0e7f-495c-9cee-96db95f344df-kube-api-access-z5qw2\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f654d9\" (UID: \"f8835928-0e7f-495c-9cee-96db95f344df\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f654d9"
Jul 13 19:26:02.601309 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:26:02.601104    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f8835928-0e7f-495c-9cee-96db95f344df-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f654d9\" (UID: \"f8835928-0e7f-495c-9cee-96db95f344df\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f654d9"
Jul 13 19:26:02.601309 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:26:02.601240    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8835928-0e7f-495c-9cee-96db95f344df-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f654d9\" (UID: \"f8835928-0e7f-495c-9cee-96db95f344df\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f654d9"
Jul 13 19:26:02.601309 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:26:02.601289    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5136b300-bb16-4ca4-a456-bc183d99dca9-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-84599bcd6b4vvr6\" (UID: \"5136b300-bb16-4ca4-a456-bc183d99dca9\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-84599bcd6b4vvr6"
Jul 13 19:26:02.603463 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:26:02.601978    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8835928-0e7f-495c-9cee-96db95f344df-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f654d9\" (UID: \"f8835928-0e7f-495c-9cee-96db95f344df\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f654d9"
Jul 13 19:26:02.603463 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:26:02.601247    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5136b300-bb16-4ca4-a456-bc183d99dca9-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-84599bcd6b4vvr6\" (UID: \"5136b300-bb16-4ca4-a456-bc183d99dca9\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-84599bcd6b4vvr6"
Jul 13 19:26:02.603463 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:26:02.602124    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5136b300-bb16-4ca4-a456-bc183d99dca9-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-84599bcd6b4vvr6\" (UID: \"5136b300-bb16-4ca4-a456-bc183d99dca9\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-84599bcd6b4vvr6"
Jul 13 19:26:02.603463 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:26:02.602283    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f8835928-0e7f-495c-9cee-96db95f344df-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f654d9\" (UID: \"f8835928-0e7f-495c-9cee-96db95f344df\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f654d9"
Jul 13 19:26:02.603463 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:26:02.602558    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f8835928-0e7f-495c-9cee-96db95f344df-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f654d9\" (UID: \"f8835928-0e7f-495c-9cee-96db95f344df\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f654d9"
Jul 13 19:26:02.603463 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:26:02.603143    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5136b300-bb16-4ca4-a456-bc183d99dca9-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-84599bcd6b4vvr6\" (UID: \"5136b300-bb16-4ca4-a456-bc183d99dca9\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-84599bcd6b4vvr6"
Jul 13 19:26:02.610391 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:26:02.604127    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5136b300-bb16-4ca4-a456-bc183d99dca9-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-84599bcd6b4vvr6\" (UID: \"5136b300-bb16-4ca4-a456-bc183d99dca9\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-84599bcd6b4vvr6"
Jul 13 19:26:02.610391 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:26:02.604567    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f8835928-0e7f-495c-9cee-96db95f344df-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f654d9\" (UID: \"f8835928-0e7f-495c-9cee-96db95f344df\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f654d9"
Jul 13 19:26:02.610391 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:26:02.605047    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8835928-0e7f-495c-9cee-96db95f344df-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f654d9\" (UID: \"f8835928-0e7f-495c-9cee-96db95f344df\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f654d9"
Jul 13 19:26:02.611405 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:26:02.611380    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5136b300-bb16-4ca4-a456-bc183d99dca9-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-84599bcd6b4vvr6\" (UID: \"5136b300-bb16-4ca4-a456-bc183d99dca9\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-84599bcd6b4vvr6"
Jul 13 19:26:02.613843 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:26:02.613818    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mj4xt\" (UniqueName: \"kubernetes.io/projected/5136b300-bb16-4ca4-a456-bc183d99dca9-kube-api-access-mj4xt\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-84599bcd6b4vvr6\" (UID: \"5136b300-bb16-4ca4-a456-bc183d99dca9\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-84599bcd6b4vvr6"
Jul 13 19:26:02.616771 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:26:02.616749    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5qw2\" (UniqueName: \"kubernetes.io/projected/f8835928-0e7f-495c-9cee-96db95f344df-kube-api-access-z5qw2\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f654d9\" (UID: \"f8835928-0e7f-495c-9cee-96db95f344df\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f654d9"
Jul 13 19:26:02.686496 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:26:02.686397    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-84599bcd6b4vvr6"
Jul 13 19:26:02.712121 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:26:02.712093    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f654d9"
Jul 13 19:26:02.848843 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:26:02.848817    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-84599bcd6b4vvr6"]
Jul 13 19:26:02.851444 ip-10-0-128-215 kubenswrapper[2565]: W0713 19:26:02.851418    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5136b300_bb16_4ca4_a456_bc183d99dca9.slice/crio-0b7b692971bb6021b347e684ae524a50f93663f6b64350c1929648fe766c18c9 WatchSource:0}: Error finding container 0b7b692971bb6021b347e684ae524a50f93663f6b64350c1929648fe766c18c9: Status 404 returned error can't find the container with id 0b7b692971bb6021b347e684ae524a50f93663f6b64350c1929648fe766c18c9
Jul 13 19:26:02.853837 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:26:02.853819    2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 13 19:26:02.875808 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:26:02.875787    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f654d9"]
Jul 13 19:26:02.877472 ip-10-0-128-215 kubenswrapper[2565]: W0713 19:26:02.877432    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8835928_0e7f_495c_9cee_96db95f344df.slice/crio-8bb78ec2a9ae8d16201463727b038329d9fd841354d1598f0205791f4beef995 WatchSource:0}: Error finding container 8bb78ec2a9ae8d16201463727b038329d9fd841354d1598f0205791f4beef995: Status 404 returned error can't find the container with id 8bb78ec2a9ae8d16201463727b038329d9fd841354d1598f0205791f4beef995
Jul 13 19:26:03.165649 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:26:03.165605    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-84599bcd6b4vvr6" event={"ID":"5136b300-bb16-4ca4-a456-bc183d99dca9","Type":"ContainerStarted","Data":"0b7b692971bb6021b347e684ae524a50f93663f6b64350c1929648fe766c18c9"}
Jul 13 19:26:03.167428 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:26:03.167387    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f654d9" event={"ID":"f8835928-0e7f-495c-9cee-96db95f344df","Type":"ContainerStarted","Data":"5c12f1c5694363597149c6cc379b275b526a1d8a1b23ea01100da51dd497d995"}
Jul 13 19:26:03.167550 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:26:03.167442    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f654d9" event={"ID":"f8835928-0e7f-495c-9cee-96db95f344df","Type":"ContainerStarted","Data":"8bb78ec2a9ae8d16201463727b038329d9fd841354d1598f0205791f4beef995"}
Jul 13 19:26:04.174692 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:26:04.174651    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-84599bcd6b4vvr6" event={"ID":"5136b300-bb16-4ca4-a456-bc183d99dca9","Type":"ContainerStarted","Data":"279b3bf082d80b9766988ae1f6ea42f103a18c31cc14902ace42e98b597d3c45"}
Jul 13 19:26:04.175171 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:26:04.174943    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-84599bcd6b4vvr6"
Jul 13 19:26:05.181443 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:26:05.181408    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-84599bcd6b4vvr6" event={"ID":"5136b300-bb16-4ca4-a456-bc183d99dca9","Type":"ContainerStarted","Data":"2c0d9124668c03cbfdce6584d5c5aab2bcb0a46d846e8d35224391920de19c2f"}
Jul 13 19:26:05.559775 ip-10-0-128-215 kubenswrapper[2565]: E0713 19:26:05.559688    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dmww7f" podUID="634b78c4-751c-4880-9f6c-1ea69b737987"
Jul 13 19:26:06.188279 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:26:06.188192    2565 generic.go:356] "Generic (PLEG): container finished" podID="f8835928-0e7f-495c-9cee-96db95f344df" containerID="5c12f1c5694363597149c6cc379b275b526a1d8a1b23ea01100da51dd497d995" exitCode=0
Jul 13 19:26:06.188279 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:26:06.188262    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f654d9" event={"ID":"f8835928-0e7f-495c-9cee-96db95f344df","Type":"ContainerDied","Data":"5c12f1c5694363597149c6cc379b275b526a1d8a1b23ea01100da51dd497d995"}
Jul 13 19:26:06.188804 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:26:06.188297    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dmww7f"
Jul 13 19:26:07.196790 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:26:07.196754    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f654d9" event={"ID":"f8835928-0e7f-495c-9cee-96db95f344df","Type":"ContainerStarted","Data":"be07a7afaa6ae47d7f431ae4420a1c0bb75ee25ac142c8791db4a2bb1f9d8720"}
Jul 13 19:26:07.226234 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:26:07.226166    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f654d9" podStartSLOduration=2.251244063 podStartE2EDuration="5.226143491s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.974899428s" isStatefulPod=false podCreationTimestamp="2026-07-13 19:26:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 19:26:07.222739936 +0000 UTC m=+2153.747873370" watchObservedRunningTime="2026-07-13 19:26:07.226143491 +0000 UTC m=+2153.751276973"
Jul 13 19:26:08.203320 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:26:08.203285    2565 generic.go:356] "Generic (PLEG): container finished" podID="5136b300-bb16-4ca4-a456-bc183d99dca9" containerID="2c0d9124668c03cbfdce6584d5c5aab2bcb0a46d846e8d35224391920de19c2f" exitCode=0
Jul 13 19:26:08.203320 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:26:08.203326    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-84599bcd6b4vvr6" event={"ID":"5136b300-bb16-4ca4-a456-bc183d99dca9","Type":"ContainerDied","Data":"2c0d9124668c03cbfdce6584d5c5aab2bcb0a46d846e8d35224391920de19c2f"}
Jul 13 19:26:09.210474 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:26:09.210431    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-84599bcd6b4vvr6" event={"ID":"5136b300-bb16-4ca4-a456-bc183d99dca9","Type":"ContainerStarted","Data":"1811307a4d11b3b4d2c1e81c048ef8379566113c0a9796958043c32be32ffddf"}
Jul 13 19:26:09.239460 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:26:09.239394    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-84599bcd6b4vvr6" podStartSLOduration=3.380476511 podStartE2EDuration="7.239375768s" totalImagesPullingTime="778.322683ms" totalInitContainerRuntime="3.080576574s" isStatefulPod=false podCreationTimestamp="2026-07-13 19:26:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 19:26:09.237731084 +0000 UTC m=+2155.762864518" watchObservedRunningTime="2026-07-13 19:26:09.239375768 +0000 UTC m=+2155.764509185"
Jul 13 19:26:12.687604 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:26:12.687552    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-84599bcd6b4vvr6"
Jul 13 19:26:12.688137 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:26:12.687629    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-84599bcd6b4vvr6"
Jul 13 19:26:12.689207 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:26:12.689176    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-84599bcd6b4vvr6" podUID="5136b300-bb16-4ca4-a456-bc183d99dca9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8001/health\": dial tcp 10.134.0.53:8001: connect: connection refused"
Jul 13 19:26:12.712308 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:26:12.712279    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f654d9"
Jul 13 19:26:12.712462 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:26:12.712340    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f654d9"
Jul 13 19:26:12.713641 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:26:12.713606    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f654d9" podUID="f8835928-0e7f-495c-9cee-96db95f344df" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": dial tcp 10.134.0.54:8000: connect: connection refused"
Jul 13 19:26:14.185679 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:26:14.185649    2565 scope.go:117] "RemoveContainer" containerID="f5dc0f57f4a882fba1d322373cb51ca6970ca28b6b75a3c3abf6fdea29d6ecc6"
Jul 13 19:26:14.197974 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:26:14.197952    2565 scope.go:117] "RemoveContainer" containerID="650bb4ed159e3b7000535911303b8768d46d6bb59dfc6e3ae186190091c88f8d"
Jul 13 19:26:22.687081 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:26:22.687025    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-84599bcd6b4vvr6" podUID="5136b300-bb16-4ca4-a456-bc183d99dca9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8001/health\": dial tcp 10.134.0.53:8001: connect: connection refused"
Jul 13 19:26:22.700249 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:26:22.700224    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-84599bcd6b4vvr6"
Jul 13 19:26:22.713172 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:26:22.713143    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f654d9" podUID="f8835928-0e7f-495c-9cee-96db95f344df" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": dial tcp 10.134.0.54:8000: connect: connection refused"
Jul 13 19:26:32.687077 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:26:32.687032    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-84599bcd6b4vvr6" podUID="5136b300-bb16-4ca4-a456-bc183d99dca9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8001/health\": dial tcp 10.134.0.53:8001: connect: connection refused"
Jul 13 19:26:32.712719 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:26:32.712652    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f654d9" podUID="f8835928-0e7f-495c-9cee-96db95f344df" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": dial tcp 10.134.0.54:8000: connect: connection refused"
Jul 13 19:26:42.687488 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:26:42.687426    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-84599bcd6b4vvr6" podUID="5136b300-bb16-4ca4-a456-bc183d99dca9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8001/health\": dial tcp 10.134.0.53:8001: connect: connection refused"
Jul 13 19:26:42.713613 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:26:42.713557    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f654d9" podUID="f8835928-0e7f-495c-9cee-96db95f344df" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": dial tcp 10.134.0.54:8000: connect: connection refused"
Jul 13 19:26:52.687515 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:26:52.687458    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-84599bcd6b4vvr6" podUID="5136b300-bb16-4ca4-a456-bc183d99dca9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8001/health\": dial tcp 10.134.0.53:8001: connect: connection refused"
Jul 13 19:26:52.713477 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:26:52.713434    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f654d9" podUID="f8835928-0e7f-495c-9cee-96db95f344df" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": dial tcp 10.134.0.54:8000: connect: connection refused"
Jul 13 19:27:02.687133 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:27:02.687084    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-84599bcd6b4vvr6" podUID="5136b300-bb16-4ca4-a456-bc183d99dca9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8001/health\": dial tcp 10.134.0.53:8001: connect: connection refused"
Jul 13 19:27:02.713095 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:27:02.713051    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f654d9" podUID="f8835928-0e7f-495c-9cee-96db95f344df" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": dial tcp 10.134.0.54:8000: connect: connection refused"
Jul 13 19:27:12.687135 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:27:12.687090    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-84599bcd6b4vvr6" podUID="5136b300-bb16-4ca4-a456-bc183d99dca9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8001/health\": dial tcp 10.134.0.53:8001: connect: connection refused"
Jul 13 19:27:12.713224 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:27:12.713190    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f654d9" podUID="f8835928-0e7f-495c-9cee-96db95f344df" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": dial tcp 10.134.0.54:8000: connect: connection refused"
Jul 13 19:27:22.687308 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:27:22.687256    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-84599bcd6b4vvr6" podUID="5136b300-bb16-4ca4-a456-bc183d99dca9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8001/health\": dial tcp 10.134.0.53:8001: connect: connection refused"
Jul 13 19:27:22.712955 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:27:22.712920    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f654d9" podUID="f8835928-0e7f-495c-9cee-96db95f344df" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": dial tcp 10.134.0.54:8000: connect: connection refused"
Jul 13 19:27:32.687276 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:27:32.687226    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-84599bcd6b4vvr6" podUID="5136b300-bb16-4ca4-a456-bc183d99dca9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8001/health\": dial tcp 10.134.0.53:8001: connect: connection refused"
Jul 13 19:27:32.712696 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:27:32.712650    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f654d9" podUID="f8835928-0e7f-495c-9cee-96db95f344df" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": dial tcp 10.134.0.54:8000: connect: connection refused"
Jul 13 19:27:42.687674 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:27:42.687559    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-84599bcd6b4vvr6" podUID="5136b300-bb16-4ca4-a456-bc183d99dca9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8001/health\": dial tcp 10.134.0.53:8001: connect: connection refused"
Jul 13 19:27:42.713718 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:27:42.713660    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f654d9" podUID="f8835928-0e7f-495c-9cee-96db95f344df" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": dial tcp 10.134.0.54:8000: connect: connection refused"
Jul 13 19:27:52.687867 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:27:52.687821    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-84599bcd6b4vvr6" podUID="5136b300-bb16-4ca4-a456-bc183d99dca9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8001/health\": dial tcp 10.134.0.53:8001: connect: connection refused"
Jul 13 19:27:52.713355 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:27:52.713324    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f654d9" podUID="f8835928-0e7f-495c-9cee-96db95f344df" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": dial tcp 10.134.0.54:8000: connect: connection refused"
Jul 13 19:28:02.686954 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:28:02.686904    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-84599bcd6b4vvr6" podUID="5136b300-bb16-4ca4-a456-bc183d99dca9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8001/health\": dial tcp 10.134.0.53:8001: connect: connection refused"
Jul 13 19:28:02.713110 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:28:02.713071    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f654d9" podUID="f8835928-0e7f-495c-9cee-96db95f344df" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": dial tcp 10.134.0.54:8000: connect: connection refused"
Jul 13 19:28:09.189426 ip-10-0-128-215 kubenswrapper[2565]: E0713 19:28:09.189387    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dmww7f" podUID="634b78c4-751c-4880-9f6c-1ea69b737987"
Jul 13 19:28:09.884767 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:28:09.884733    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dmww7f"
Jul 13 19:28:12.686920 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:28:12.686877    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-84599bcd6b4vvr6" podUID="5136b300-bb16-4ca4-a456-bc183d99dca9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8001/health\": dial tcp 10.134.0.53:8001: connect: connection refused"
Jul 13 19:28:12.713230 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:28:12.713189    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f654d9" podUID="f8835928-0e7f-495c-9cee-96db95f344df" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": dial tcp 10.134.0.54:8000: connect: connection refused"
Jul 13 19:28:22.697590 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:28:22.697547    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-84599bcd6b4vvr6"
Jul 13 19:28:22.710184 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:28:22.710162    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-84599bcd6b4vvr6"
Jul 13 19:28:22.713417 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:28:22.713390    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f654d9" podUID="f8835928-0e7f-495c-9cee-96db95f344df" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": dial tcp 10.134.0.54:8000: connect: connection refused"
Jul 13 19:28:32.712571 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:28:32.712530    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f654d9" podUID="f8835928-0e7f-495c-9cee-96db95f344df" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": dial tcp 10.134.0.54:8000: connect: connection refused"
Jul 13 19:28:42.722258 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:28:42.722229    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f654d9"
Jul 13 19:28:42.729857 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:28:42.729836    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f654d9"
Jul 13 19:28:49.451368 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:28:49.451335    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-84599bcd6b4vvr6"]
Jul 13 19:28:49.471728 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:28:49.469488    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f654d9"]
Jul 13 19:29:04.486769 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:29:04.486600    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-84599bcd6b4vvr6" podUID="5136b300-bb16-4ca4-a456-bc183d99dca9" containerName="main" containerID="cri-o://1811307a4d11b3b4d2c1e81c048ef8379566113c0a9796958043c32be32ffddf" gracePeriod=45
Jul 13 19:29:04.500500 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:29:04.500455    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f654d9" podUID="f8835928-0e7f-495c-9cee-96db95f344df" containerName="main" containerID="cri-o://be07a7afaa6ae47d7f431ae4420a1c0bb75ee25ac142c8791db4a2bb1f9d8720" gracePeriod=45
Jul 13 19:29:05.307112 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:29:05.307060    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-84599bcd6b4vvr6" podUID="5136b300-bb16-4ca4-a456-bc183d99dca9" containerName="llm-d-routing-sidecar" containerID="cri-o://279b3bf082d80b9766988ae1f6ea42f103a18c31cc14902ace42e98b597d3c45" gracePeriod=45
Jul 13 19:29:05.440745 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:29:05.440719    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f654d9"
Jul 13 19:29:05.536026 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:29:05.535994    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8835928-0e7f-495c-9cee-96db95f344df-model-cache\") pod \"f8835928-0e7f-495c-9cee-96db95f344df\" (UID: \"f8835928-0e7f-495c-9cee-96db95f344df\") "
Jul 13 19:29:05.536462 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:29:05.536037    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8835928-0e7f-495c-9cee-96db95f344df-tls-certs\") pod \"f8835928-0e7f-495c-9cee-96db95f344df\" (UID: \"f8835928-0e7f-495c-9cee-96db95f344df\") "
Jul 13 19:29:05.536462 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:29:05.536072    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f8835928-0e7f-495c-9cee-96db95f344df-dshm\") pod \"f8835928-0e7f-495c-9cee-96db95f344df\" (UID: \"f8835928-0e7f-495c-9cee-96db95f344df\") "
Jul 13 19:29:05.536462 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:29:05.536110    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z5qw2\" (UniqueName: \"kubernetes.io/projected/f8835928-0e7f-495c-9cee-96db95f344df-kube-api-access-z5qw2\") pod \"f8835928-0e7f-495c-9cee-96db95f344df\" (UID: \"f8835928-0e7f-495c-9cee-96db95f344df\") "
Jul 13 19:29:05.536462 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:29:05.536185    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f8835928-0e7f-495c-9cee-96db95f344df-kserve-provision-location\") pod \"f8835928-0e7f-495c-9cee-96db95f344df\" (UID: \"f8835928-0e7f-495c-9cee-96db95f344df\") "
Jul 13 19:29:05.536462 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:29:05.536206    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f8835928-0e7f-495c-9cee-96db95f344df-home\") pod \"f8835928-0e7f-495c-9cee-96db95f344df\" (UID: \"f8835928-0e7f-495c-9cee-96db95f344df\") "
Jul 13 19:29:05.536462 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:29:05.536236    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8835928-0e7f-495c-9cee-96db95f344df-tmp-dir\") pod \"f8835928-0e7f-495c-9cee-96db95f344df\" (UID: \"f8835928-0e7f-495c-9cee-96db95f344df\") "
Jul 13 19:29:05.536462 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:29:05.536297    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8835928-0e7f-495c-9cee-96db95f344df-model-cache" (OuterVolumeSpecName: "model-cache") pod "f8835928-0e7f-495c-9cee-96db95f344df" (UID: "f8835928-0e7f-495c-9cee-96db95f344df"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:29:05.536883 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:29:05.536555    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8835928-0e7f-495c-9cee-96db95f344df-model-cache\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 13 19:29:05.537183 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:29:05.537158    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8835928-0e7f-495c-9cee-96db95f344df-home" (OuterVolumeSpecName: "home") pod "f8835928-0e7f-495c-9cee-96db95f344df" (UID: "f8835928-0e7f-495c-9cee-96db95f344df"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:29:05.539907 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:29:05.539881    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8835928-0e7f-495c-9cee-96db95f344df-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f8835928-0e7f-495c-9cee-96db95f344df" (UID: "f8835928-0e7f-495c-9cee-96db95f344df"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 19:29:05.540049 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:29:05.540034    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8835928-0e7f-495c-9cee-96db95f344df-dshm" (OuterVolumeSpecName: "dshm") pod "f8835928-0e7f-495c-9cee-96db95f344df" (UID: "f8835928-0e7f-495c-9cee-96db95f344df"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:29:05.540110 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:29:05.540081    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8835928-0e7f-495c-9cee-96db95f344df-kube-api-access-z5qw2" (OuterVolumeSpecName: "kube-api-access-z5qw2") pod "f8835928-0e7f-495c-9cee-96db95f344df" (UID: "f8835928-0e7f-495c-9cee-96db95f344df"). InnerVolumeSpecName "kube-api-access-z5qw2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 19:29:05.549695 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:29:05.549671    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8835928-0e7f-495c-9cee-96db95f344df-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f8835928-0e7f-495c-9cee-96db95f344df" (UID: "f8835928-0e7f-495c-9cee-96db95f344df"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:29:05.558897 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:29:05.558877    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-84599bcd6b4vvr6"
Jul 13 19:29:05.607129 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:29:05.607076    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8835928-0e7f-495c-9cee-96db95f344df-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f8835928-0e7f-495c-9cee-96db95f344df" (UID: "f8835928-0e7f-495c-9cee-96db95f344df"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:29:05.636850 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:29:05.636824    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5136b300-bb16-4ca4-a456-bc183d99dca9-model-cache\") pod \"5136b300-bb16-4ca4-a456-bc183d99dca9\" (UID: \"5136b300-bb16-4ca4-a456-bc183d99dca9\") "
Jul 13 19:29:05.636971 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:29:05.636871    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mj4xt\" (UniqueName: \"kubernetes.io/projected/5136b300-bb16-4ca4-a456-bc183d99dca9-kube-api-access-mj4xt\") pod \"5136b300-bb16-4ca4-a456-bc183d99dca9\" (UID: \"5136b300-bb16-4ca4-a456-bc183d99dca9\") "
Jul 13 19:29:05.636971 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:29:05.636906    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5136b300-bb16-4ca4-a456-bc183d99dca9-tmp-dir\") pod \"5136b300-bb16-4ca4-a456-bc183d99dca9\" (UID: \"5136b300-bb16-4ca4-a456-bc183d99dca9\") "
Jul 13 19:29:05.636971 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:29:05.636939    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5136b300-bb16-4ca4-a456-bc183d99dca9-kserve-provision-location\") pod \"5136b300-bb16-4ca4-a456-bc183d99dca9\" (UID: \"5136b300-bb16-4ca4-a456-bc183d99dca9\") "
Jul 13 19:29:05.636971 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:29:05.636962    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5136b300-bb16-4ca4-a456-bc183d99dca9-tls-certs\") pod \"5136b300-bb16-4ca4-a456-bc183d99dca9\" (UID: \"5136b300-bb16-4ca4-a456-bc183d99dca9\") "
Jul 13 19:29:05.637183 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:29:05.636983    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5136b300-bb16-4ca4-a456-bc183d99dca9-home\") pod \"5136b300-bb16-4ca4-a456-bc183d99dca9\" (UID: \"5136b300-bb16-4ca4-a456-bc183d99dca9\") "
Jul 13 19:29:05.637183 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:29:05.637002    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5136b300-bb16-4ca4-a456-bc183d99dca9-dshm\") pod \"5136b300-bb16-4ca4-a456-bc183d99dca9\" (UID: \"5136b300-bb16-4ca4-a456-bc183d99dca9\") "
Jul 13 19:29:05.637183 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:29:05.637083    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5136b300-bb16-4ca4-a456-bc183d99dca9-model-cache" (OuterVolumeSpecName: "model-cache") pod "5136b300-bb16-4ca4-a456-bc183d99dca9" (UID: "5136b300-bb16-4ca4-a456-bc183d99dca9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:29:05.637373 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:29:05.637353    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f8835928-0e7f-495c-9cee-96db95f344df-kserve-provision-location\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 13 19:29:05.637439 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:29:05.637378    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f8835928-0e7f-495c-9cee-96db95f344df-home\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 13 19:29:05.637439 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:29:05.637392    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8835928-0e7f-495c-9cee-96db95f344df-tmp-dir\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 13 19:29:05.637439 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:29:05.637405    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5136b300-bb16-4ca4-a456-bc183d99dca9-model-cache\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 13 19:29:05.637439 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:29:05.637416    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8835928-0e7f-495c-9cee-96db95f344df-tls-certs\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 13 19:29:05.637439 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:29:05.637427    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f8835928-0e7f-495c-9cee-96db95f344df-dshm\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 13 19:29:05.637439 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:29:05.637440    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z5qw2\" (UniqueName: \"kubernetes.io/projected/f8835928-0e7f-495c-9cee-96db95f344df-kube-api-access-z5qw2\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 13 19:29:05.637758 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:29:05.637664    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5136b300-bb16-4ca4-a456-bc183d99dca9-home" (OuterVolumeSpecName: "home") pod "5136b300-bb16-4ca4-a456-bc183d99dca9" (UID: "5136b300-bb16-4ca4-a456-bc183d99dca9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:29:05.639310 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:29:05.639289    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5136b300-bb16-4ca4-a456-bc183d99dca9-dshm" (OuterVolumeSpecName: "dshm") pod "5136b300-bb16-4ca4-a456-bc183d99dca9" (UID: "5136b300-bb16-4ca4-a456-bc183d99dca9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:29:05.639429 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:29:05.639407    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5136b300-bb16-4ca4-a456-bc183d99dca9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5136b300-bb16-4ca4-a456-bc183d99dca9" (UID: "5136b300-bb16-4ca4-a456-bc183d99dca9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 19:29:05.639576 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:29:05.639554    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5136b300-bb16-4ca4-a456-bc183d99dca9-kube-api-access-mj4xt" (OuterVolumeSpecName: "kube-api-access-mj4xt") pod "5136b300-bb16-4ca4-a456-bc183d99dca9" (UID: "5136b300-bb16-4ca4-a456-bc183d99dca9"). InnerVolumeSpecName "kube-api-access-mj4xt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 19:29:05.650206 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:29:05.650183    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5136b300-bb16-4ca4-a456-bc183d99dca9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5136b300-bb16-4ca4-a456-bc183d99dca9" (UID: "5136b300-bb16-4ca4-a456-bc183d99dca9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:29:05.696527 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:29:05.696492    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5136b300-bb16-4ca4-a456-bc183d99dca9-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5136b300-bb16-4ca4-a456-bc183d99dca9" (UID: "5136b300-bb16-4ca4-a456-bc183d99dca9"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:29:05.738811 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:29:05.738739    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mj4xt\" (UniqueName: \"kubernetes.io/projected/5136b300-bb16-4ca4-a456-bc183d99dca9-kube-api-access-mj4xt\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 13 19:29:05.738811 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:29:05.738765    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5136b300-bb16-4ca4-a456-bc183d99dca9-tmp-dir\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 13 19:29:05.738811 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:29:05.738778    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5136b300-bb16-4ca4-a456-bc183d99dca9-kserve-provision-location\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 13 19:29:05.738811 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:29:05.738786    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5136b300-bb16-4ca4-a456-bc183d99dca9-tls-certs\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 13 19:29:05.738811 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:29:05.738797    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5136b300-bb16-4ca4-a456-bc183d99dca9-home\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 13 19:29:05.738811 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:29:05.738808    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5136b300-bb16-4ca4-a456-bc183d99dca9-dshm\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 13 19:29:06.188681 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:29:06.188644    2565 generic.go:356] "Generic (PLEG): container finished" podID="5136b300-bb16-4ca4-a456-bc183d99dca9" containerID="1811307a4d11b3b4d2c1e81c048ef8379566113c0a9796958043c32be32ffddf" exitCode=0
Jul 13 19:29:06.188681 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:29:06.188673    2565 generic.go:356] "Generic (PLEG): container finished" podID="5136b300-bb16-4ca4-a456-bc183d99dca9" containerID="279b3bf082d80b9766988ae1f6ea42f103a18c31cc14902ace42e98b597d3c45" exitCode=0
Jul 13 19:29:06.189033 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:29:06.188745    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-84599bcd6b4vvr6"
Jul 13 19:29:06.189033 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:29:06.188734    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-84599bcd6b4vvr6" event={"ID":"5136b300-bb16-4ca4-a456-bc183d99dca9","Type":"ContainerDied","Data":"1811307a4d11b3b4d2c1e81c048ef8379566113c0a9796958043c32be32ffddf"}
Jul 13 19:29:06.189033 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:29:06.188857    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-84599bcd6b4vvr6" event={"ID":"5136b300-bb16-4ca4-a456-bc183d99dca9","Type":"ContainerDied","Data":"279b3bf082d80b9766988ae1f6ea42f103a18c31cc14902ace42e98b597d3c45"}
Jul 13 19:29:06.189033 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:29:06.188899    2565 scope.go:117] "RemoveContainer" containerID="1811307a4d11b3b4d2c1e81c048ef8379566113c0a9796958043c32be32ffddf"
Jul 13 19:29:06.189256 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:29:06.189242    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-84599bcd6b4vvr6" event={"ID":"5136b300-bb16-4ca4-a456-bc183d99dca9","Type":"ContainerDied","Data":"0b7b692971bb6021b347e684ae524a50f93663f6b64350c1929648fe766c18c9"}
Jul 13 19:29:06.190914 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:29:06.190888    2565 generic.go:356] "Generic (PLEG): container finished" podID="f8835928-0e7f-495c-9cee-96db95f344df" containerID="be07a7afaa6ae47d7f431ae4420a1c0bb75ee25ac142c8791db4a2bb1f9d8720" exitCode=0
Jul 13 19:29:06.191028 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:29:06.190974    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f654d9" event={"ID":"f8835928-0e7f-495c-9cee-96db95f344df","Type":"ContainerDied","Data":"be07a7afaa6ae47d7f431ae4420a1c0bb75ee25ac142c8791db4a2bb1f9d8720"}
Jul 13 19:29:06.191028 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:29:06.191017    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f654d9" event={"ID":"f8835928-0e7f-495c-9cee-96db95f344df","Type":"ContainerDied","Data":"8bb78ec2a9ae8d16201463727b038329d9fd841354d1598f0205791f4beef995"}
Jul 13 19:29:06.191121 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:29:06.191101    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f654d9"
Jul 13 19:29:06.198679 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:29:06.198667    2565 scope.go:117] "RemoveContainer" containerID="2c0d9124668c03cbfdce6584d5c5aab2bcb0a46d846e8d35224391920de19c2f"
Jul 13 19:29:06.210032 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:29:06.210007    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-84599bcd6b4vvr6"]
Jul 13 19:29:06.212531 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:29:06.212513    2565 scope.go:117] "RemoveContainer" containerID="279b3bf082d80b9766988ae1f6ea42f103a18c31cc14902ace42e98b597d3c45"
Jul 13 19:29:06.217157 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:29:06.217138    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-84599bcd6b4vvr6"]
Jul 13 19:29:06.221459 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:29:06.221442    2565 scope.go:117] "RemoveContainer" containerID="1811307a4d11b3b4d2c1e81c048ef8379566113c0a9796958043c32be32ffddf"
Jul 13 19:29:06.221741 ip-10-0-128-215 kubenswrapper[2565]: E0713 19:29:06.221721    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1811307a4d11b3b4d2c1e81c048ef8379566113c0a9796958043c32be32ffddf\": container with ID starting with 1811307a4d11b3b4d2c1e81c048ef8379566113c0a9796958043c32be32ffddf not found: ID does not exist" containerID="1811307a4d11b3b4d2c1e81c048ef8379566113c0a9796958043c32be32ffddf"
Jul 13 19:29:06.221802 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:29:06.221750    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1811307a4d11b3b4d2c1e81c048ef8379566113c0a9796958043c32be32ffddf"} err="failed to get container status \"1811307a4d11b3b4d2c1e81c048ef8379566113c0a9796958043c32be32ffddf\": rpc error: code = NotFound desc = could not find container \"1811307a4d11b3b4d2c1e81c048ef8379566113c0a9796958043c32be32ffddf\": container with ID starting with 1811307a4d11b3b4d2c1e81c048ef8379566113c0a9796958043c32be32ffddf not found: ID does not exist"
Jul 13 19:29:06.221802 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:29:06.221769    2565 scope.go:117] "RemoveContainer" containerID="2c0d9124668c03cbfdce6584d5c5aab2bcb0a46d846e8d35224391920de19c2f"
Jul 13 19:29:06.222011 ip-10-0-128-215 kubenswrapper[2565]: E0713 19:29:06.221995    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c0d9124668c03cbfdce6584d5c5aab2bcb0a46d846e8d35224391920de19c2f\": container with ID starting with 2c0d9124668c03cbfdce6584d5c5aab2bcb0a46d846e8d35224391920de19c2f not found: ID does not exist" containerID="2c0d9124668c03cbfdce6584d5c5aab2bcb0a46d846e8d35224391920de19c2f"
Jul 13 19:29:06.222064 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:29:06.222016    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c0d9124668c03cbfdce6584d5c5aab2bcb0a46d846e8d35224391920de19c2f"} err="failed to get container status \"2c0d9124668c03cbfdce6584d5c5aab2bcb0a46d846e8d35224391920de19c2f\": rpc error: code = NotFound desc = could not find container \"2c0d9124668c03cbfdce6584d5c5aab2bcb0a46d846e8d35224391920de19c2f\": container with ID starting with 2c0d9124668c03cbfdce6584d5c5aab2bcb0a46d846e8d35224391920de19c2f not found: ID does not exist"
Jul 13 19:29:06.222064 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:29:06.222030    2565 scope.go:117] "RemoveContainer" containerID="279b3bf082d80b9766988ae1f6ea42f103a18c31cc14902ace42e98b597d3c45"
Jul 13 19:29:06.222237 ip-10-0-128-215 kubenswrapper[2565]: E0713 19:29:06.222222    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"279b3bf082d80b9766988ae1f6ea42f103a18c31cc14902ace42e98b597d3c45\": container with ID starting with 279b3bf082d80b9766988ae1f6ea42f103a18c31cc14902ace42e98b597d3c45 not found: ID does not exist" containerID="279b3bf082d80b9766988ae1f6ea42f103a18c31cc14902ace42e98b597d3c45"
Jul 13 19:29:06.222281 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:29:06.222241    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"279b3bf082d80b9766988ae1f6ea42f103a18c31cc14902ace42e98b597d3c45"} err="failed to get container status \"279b3bf082d80b9766988ae1f6ea42f103a18c31cc14902ace42e98b597d3c45\": rpc error: code = NotFound desc = could not find container \"279b3bf082d80b9766988ae1f6ea42f103a18c31cc14902ace42e98b597d3c45\": container with ID starting with 279b3bf082d80b9766988ae1f6ea42f103a18c31cc14902ace42e98b597d3c45 not found: ID does not exist"
Jul 13 19:29:06.222281 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:29:06.222253    2565 scope.go:117] "RemoveContainer" containerID="1811307a4d11b3b4d2c1e81c048ef8379566113c0a9796958043c32be32ffddf"
Jul 13 19:29:06.222444 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:29:06.222428    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1811307a4d11b3b4d2c1e81c048ef8379566113c0a9796958043c32be32ffddf"} err="failed to get container status \"1811307a4d11b3b4d2c1e81c048ef8379566113c0a9796958043c32be32ffddf\": rpc error: code = NotFound desc = could not find container \"1811307a4d11b3b4d2c1e81c048ef8379566113c0a9796958043c32be32ffddf\": container with ID starting with 1811307a4d11b3b4d2c1e81c048ef8379566113c0a9796958043c32be32ffddf not found: ID does not exist"
Jul 13 19:29:06.222492 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:29:06.222444    2565 scope.go:117] "RemoveContainer" containerID="2c0d9124668c03cbfdce6584d5c5aab2bcb0a46d846e8d35224391920de19c2f"
Jul 13 19:29:06.222650 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:29:06.222629    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c0d9124668c03cbfdce6584d5c5aab2bcb0a46d846e8d35224391920de19c2f"} err="failed to get container status \"2c0d9124668c03cbfdce6584d5c5aab2bcb0a46d846e8d35224391920de19c2f\": rpc error: code = NotFound desc = could not find container \"2c0d9124668c03cbfdce6584d5c5aab2bcb0a46d846e8d35224391920de19c2f\": container with ID starting with 2c0d9124668c03cbfdce6584d5c5aab2bcb0a46d846e8d35224391920de19c2f not found: ID does not exist"
Jul 13 19:29:06.222714 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:29:06.222651    2565 scope.go:117] "RemoveContainer" containerID="279b3bf082d80b9766988ae1f6ea42f103a18c31cc14902ace42e98b597d3c45"
Jul 13 19:29:06.222892 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:29:06.222875    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"279b3bf082d80b9766988ae1f6ea42f103a18c31cc14902ace42e98b597d3c45"} err="failed to get container status \"279b3bf082d80b9766988ae1f6ea42f103a18c31cc14902ace42e98b597d3c45\": rpc error: code = NotFound desc = could not find container \"279b3bf082d80b9766988ae1f6ea42f103a18c31cc14902ace42e98b597d3c45\": container with ID starting with 279b3bf082d80b9766988ae1f6ea42f103a18c31cc14902ace42e98b597d3c45 not found: ID does not exist"
Jul 13 19:29:06.222947 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:29:06.222893    2565 scope.go:117] "RemoveContainer" containerID="be07a7afaa6ae47d7f431ae4420a1c0bb75ee25ac142c8791db4a2bb1f9d8720"
Jul 13 19:29:06.227686 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:29:06.227669    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f654d9"]
Jul 13 19:29:06.231520 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:29:06.231496    2565 scope.go:117] "RemoveContainer" containerID="5c12f1c5694363597149c6cc379b275b526a1d8a1b23ea01100da51dd497d995"
Jul 13 19:29:06.233880 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:29:06.233861    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6f654d9"]
Jul 13 19:29:06.239830 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:29:06.239815    2565 scope.go:117] "RemoveContainer" containerID="be07a7afaa6ae47d7f431ae4420a1c0bb75ee25ac142c8791db4a2bb1f9d8720"
Jul 13 19:29:06.240076 ip-10-0-128-215 kubenswrapper[2565]: E0713 19:29:06.240057    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be07a7afaa6ae47d7f431ae4420a1c0bb75ee25ac142c8791db4a2bb1f9d8720\": container with ID starting with be07a7afaa6ae47d7f431ae4420a1c0bb75ee25ac142c8791db4a2bb1f9d8720 not found: ID does not exist" containerID="be07a7afaa6ae47d7f431ae4420a1c0bb75ee25ac142c8791db4a2bb1f9d8720"
Jul 13 19:29:06.240160 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:29:06.240087    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be07a7afaa6ae47d7f431ae4420a1c0bb75ee25ac142c8791db4a2bb1f9d8720"} err="failed to get container status \"be07a7afaa6ae47d7f431ae4420a1c0bb75ee25ac142c8791db4a2bb1f9d8720\": rpc error: code = NotFound desc = could not find container \"be07a7afaa6ae47d7f431ae4420a1c0bb75ee25ac142c8791db4a2bb1f9d8720\": container with ID starting with be07a7afaa6ae47d7f431ae4420a1c0bb75ee25ac142c8791db4a2bb1f9d8720 not found: ID does not exist"
Jul 13 19:29:06.240160 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:29:06.240110    2565 scope.go:117] "RemoveContainer" containerID="5c12f1c5694363597149c6cc379b275b526a1d8a1b23ea01100da51dd497d995"
Jul 13 19:29:06.240354 ip-10-0-128-215 kubenswrapper[2565]: E0713 19:29:06.240337    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c12f1c5694363597149c6cc379b275b526a1d8a1b23ea01100da51dd497d995\": container with ID starting with 5c12f1c5694363597149c6cc379b275b526a1d8a1b23ea01100da51dd497d995 not found: ID does not exist" containerID="5c12f1c5694363597149c6cc379b275b526a1d8a1b23ea01100da51dd497d995"
Jul 13 19:29:06.240397 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:29:06.240362    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c12f1c5694363597149c6cc379b275b526a1d8a1b23ea01100da51dd497d995"} err="failed to get container status \"5c12f1c5694363597149c6cc379b275b526a1d8a1b23ea01100da51dd497d995\": rpc error: code = NotFound desc = could not find container \"5c12f1c5694363597149c6cc379b275b526a1d8a1b23ea01100da51dd497d995\": container with ID starting with 5c12f1c5694363597149c6cc379b275b526a1d8a1b23ea01100da51dd497d995 not found: ID does not exist"
Jul 13 19:29:08.048083 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:29:08.048050    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5136b300-bb16-4ca4-a456-bc183d99dca9" path="/var/lib/kubelet/pods/5136b300-bb16-4ca4-a456-bc183d99dca9/volumes"
Jul 13 19:29:08.048510 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:29:08.048497    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f8835928-0e7f-495c-9cee-96db95f344df" path="/var/lib/kubelet/pods/f8835928-0e7f-495c-9cee-96db95f344df/volumes"
Jul 13 19:29:30.974852 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:29:30.974818    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6d9d59446f-spptn"]
Jul 13 19:29:30.975405 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:29:30.975386    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8835928-0e7f-495c-9cee-96db95f344df" containerName="main"
Jul 13 19:29:30.975508 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:29:30.975409    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8835928-0e7f-495c-9cee-96db95f344df" containerName="main"
Jul 13 19:29:30.975508 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:29:30.975431    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8835928-0e7f-495c-9cee-96db95f344df" containerName="storage-initializer"
Jul 13 19:29:30.975508 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:29:30.975439    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8835928-0e7f-495c-9cee-96db95f344df" containerName="storage-initializer"
Jul 13 19:29:30.975508 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:29:30.975453    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5136b300-bb16-4ca4-a456-bc183d99dca9" containerName="llm-d-routing-sidecar"
Jul 13 19:29:30.975508 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:29:30.975461    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="5136b300-bb16-4ca4-a456-bc183d99dca9" containerName="llm-d-routing-sidecar"
Jul 13 19:29:30.975508 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:29:30.975485    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5136b300-bb16-4ca4-a456-bc183d99dca9" containerName="main"
Jul 13 19:29:30.975508 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:29:30.975493    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="5136b300-bb16-4ca4-a456-bc183d99dca9" containerName="main"
Jul 13 19:29:30.975867 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:29:30.975515    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5136b300-bb16-4ca4-a456-bc183d99dca9" containerName="storage-initializer"
Jul 13 19:29:30.975867 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:29:30.975524    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="5136b300-bb16-4ca4-a456-bc183d99dca9" containerName="storage-initializer"
Jul 13 19:29:30.975867 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:29:30.975604    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8835928-0e7f-495c-9cee-96db95f344df" containerName="main"
Jul 13 19:29:30.975867 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:29:30.975620    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="5136b300-bb16-4ca4-a456-bc183d99dca9" containerName="llm-d-routing-sidecar"
Jul 13 19:29:30.975867 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:29:30.975632    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="5136b300-bb16-4ca4-a456-bc183d99dca9" containerName="main"
Jul 13 19:29:30.978147 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:29:30.978127    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6d9d59446f-spptn"
Jul 13 19:29:30.982759 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:29:30.982310    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 13 19:29:30.983925 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:29:30.983905    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"custom-route-timeout-pd-test-kserve-self-signed-certs\""
Jul 13 19:29:30.984046 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:29:30.983982    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"custom-route-timeout-pd-test-kserve-dockercfg-wtfwc\""
Jul 13 19:29:30.985167 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:29:30.985149    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-f7f58c4f6-nsctq"]
Jul 13 19:29:30.988020 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:29:30.988005    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-f7f58c4f6-nsctq"
Jul 13 19:29:30.995623 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:29:30.995605    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6d9d59446f-spptn"]
Jul 13 19:29:31.004668 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:29:31.004642    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-f7f58c4f6-nsctq"]
Jul 13 19:29:31.053161 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:29:31.053138    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62328ddb-2423-4182-9a77-b0d7c284f208-dshm\") pod \"custom-route-timeout-pd-test-kserve-6d9d59446f-spptn\" (UID: \"62328ddb-2423-4182-9a77-b0d7c284f208\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6d9d59446f-spptn"
Jul 13 19:29:31.053268 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:29:31.053183    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62328ddb-2423-4182-9a77-b0d7c284f208-home\") pod \"custom-route-timeout-pd-test-kserve-6d9d59446f-spptn\" (UID: \"62328ddb-2423-4182-9a77-b0d7c284f208\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6d9d59446f-spptn"
Jul 13 19:29:31.053268 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:29:31.053212    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f5a7e70-d1ae-4938-bd11-e286e8bac45a-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-prefill-f7f58c4f6-nsctq\" (UID: \"6f5a7e70-d1ae-4938-bd11-e286e8bac45a\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-f7f58c4f6-nsctq"
Jul 13 19:29:31.053268 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:29:31.053234    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f5a7e70-d1ae-4938-bd11-e286e8bac45a-home\") pod \"custom-route-timeout-pd-test-kserve-prefill-f7f58c4f6-nsctq\" (UID: \"6f5a7e70-d1ae-4938-bd11-e286e8bac45a\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-f7f58c4f6-nsctq"
Jul 13 19:29:31.053268 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:29:31.053252    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6f5a7e70-d1ae-4938-bd11-e286e8bac45a-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-prefill-f7f58c4f6-nsctq\" (UID: \"6f5a7e70-d1ae-4938-bd11-e286e8bac45a\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-f7f58c4f6-nsctq"
Jul 13 19:29:31.053426 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:29:31.053269    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62328ddb-2423-4182-9a77-b0d7c284f208-model-cache\") pod \"custom-route-timeout-pd-test-kserve-6d9d59446f-spptn\" (UID: \"62328ddb-2423-4182-9a77-b0d7c284f208\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6d9d59446f-spptn"
Jul 13 19:29:31.053426 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:29:31.053304    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzq2p\" (UniqueName: \"kubernetes.io/projected/62328ddb-2423-4182-9a77-b0d7c284f208-kube-api-access-tzq2p\") pod \"custom-route-timeout-pd-test-kserve-6d9d59446f-spptn\" (UID: \"62328ddb-2423-4182-9a77-b0d7c284f208\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6d9d59446f-spptn"
Jul 13 19:29:31.053426 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:29:31.053339    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxwl6\" (UniqueName: \"kubernetes.io/projected/6f5a7e70-d1ae-4938-bd11-e286e8bac45a-kube-api-access-dxwl6\") pod \"custom-route-timeout-pd-test-kserve-prefill-f7f58c4f6-nsctq\" (UID: \"6f5a7e70-d1ae-4938-bd11-e286e8bac45a\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-f7f58c4f6-nsctq"
Jul 13 19:29:31.053426 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:29:31.053356    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62328ddb-2423-4182-9a77-b0d7c284f208-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-6d9d59446f-spptn\" (UID: \"62328ddb-2423-4182-9a77-b0d7c284f208\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6d9d59446f-spptn"
Jul 13 19:29:31.053426 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:29:31.053379    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f5a7e70-d1ae-4938-bd11-e286e8bac45a-model-cache\") pod \"custom-route-timeout-pd-test-kserve-prefill-f7f58c4f6-nsctq\" (UID: \"6f5a7e70-d1ae-4938-bd11-e286e8bac45a\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-f7f58c4f6-nsctq"
Jul 13 19:29:31.053426 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:29:31.053419    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62328ddb-2423-4182-9a77-b0d7c284f208-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-6d9d59446f-spptn\" (UID: \"62328ddb-2423-4182-9a77-b0d7c284f208\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6d9d59446f-spptn"
Jul 13 19:29:31.053605 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:29:31.053454    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f5a7e70-d1ae-4938-bd11-e286e8bac45a-dshm\") pod \"custom-route-timeout-pd-test-kserve-prefill-f7f58c4f6-nsctq\" (UID: \"6f5a7e70-d1ae-4938-bd11-e286e8bac45a\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-f7f58c4f6-nsctq"
Jul 13 19:29:31.053605 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:29:31.053485    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f5a7e70-d1ae-4938-bd11-e286e8bac45a-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-prefill-f7f58c4f6-nsctq\" (UID: \"6f5a7e70-d1ae-4938-bd11-e286e8bac45a\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-f7f58c4f6-nsctq"
Jul 13 19:29:31.053605 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:29:31.053509    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/62328ddb-2423-4182-9a77-b0d7c284f208-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-6d9d59446f-spptn\" (UID: \"62328ddb-2423-4182-9a77-b0d7c284f208\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6d9d59446f-spptn"
Jul 13 19:29:31.154589 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:29:31.154558    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62328ddb-2423-4182-9a77-b0d7c284f208-dshm\") pod \"custom-route-timeout-pd-test-kserve-6d9d59446f-spptn\" (UID: \"62328ddb-2423-4182-9a77-b0d7c284f208\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6d9d59446f-spptn"
Jul 13 19:29:31.154738 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:29:31.154611    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62328ddb-2423-4182-9a77-b0d7c284f208-home\") pod \"custom-route-timeout-pd-test-kserve-6d9d59446f-spptn\" (UID: \"62328ddb-2423-4182-9a77-b0d7c284f208\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6d9d59446f-spptn"
Jul 13 19:29:31.154738 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:29:31.154638    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f5a7e70-d1ae-4938-bd11-e286e8bac45a-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-prefill-f7f58c4f6-nsctq\" (UID: \"6f5a7e70-d1ae-4938-bd11-e286e8bac45a\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-f7f58c4f6-nsctq"
Jul 13 19:29:31.154738 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:29:31.154664    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f5a7e70-d1ae-4938-bd11-e286e8bac45a-home\") pod \"custom-route-timeout-pd-test-kserve-prefill-f7f58c4f6-nsctq\" (UID: \"6f5a7e70-d1ae-4938-bd11-e286e8bac45a\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-f7f58c4f6-nsctq"
Jul 13 19:29:31.154738 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:29:31.154687    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6f5a7e70-d1ae-4938-bd11-e286e8bac45a-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-prefill-f7f58c4f6-nsctq\" (UID: \"6f5a7e70-d1ae-4938-bd11-e286e8bac45a\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-f7f58c4f6-nsctq"
Jul 13 19:29:31.154738 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:29:31.154737    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62328ddb-2423-4182-9a77-b0d7c284f208-model-cache\") pod \"custom-route-timeout-pd-test-kserve-6d9d59446f-spptn\" (UID: \"62328ddb-2423-4182-9a77-b0d7c284f208\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6d9d59446f-spptn"
Jul 13 19:29:31.155003 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:29:31.154773    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzq2p\" (UniqueName: \"kubernetes.io/projected/62328ddb-2423-4182-9a77-b0d7c284f208-kube-api-access-tzq2p\") pod \"custom-route-timeout-pd-test-kserve-6d9d59446f-spptn\" (UID: \"62328ddb-2423-4182-9a77-b0d7c284f208\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6d9d59446f-spptn"
Jul 13 19:29:31.155003 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:29:31.154816    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxwl6\" (UniqueName: \"kubernetes.io/projected/6f5a7e70-d1ae-4938-bd11-e286e8bac45a-kube-api-access-dxwl6\") pod \"custom-route-timeout-pd-test-kserve-prefill-f7f58c4f6-nsctq\" (UID: \"6f5a7e70-d1ae-4938-bd11-e286e8bac45a\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-f7f58c4f6-nsctq"
Jul 13 19:29:31.155003 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:29:31.154841    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62328ddb-2423-4182-9a77-b0d7c284f208-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-6d9d59446f-spptn\" (UID: \"62328ddb-2423-4182-9a77-b0d7c284f208\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6d9d59446f-spptn"
Jul 13 19:29:31.155003 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:29:31.154864    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f5a7e70-d1ae-4938-bd11-e286e8bac45a-model-cache\") pod \"custom-route-timeout-pd-test-kserve-prefill-f7f58c4f6-nsctq\" (UID: \"6f5a7e70-d1ae-4938-bd11-e286e8bac45a\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-f7f58c4f6-nsctq"
Jul 13 19:29:31.155003 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:29:31.154920    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62328ddb-2423-4182-9a77-b0d7c284f208-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-6d9d59446f-spptn\" (UID: \"62328ddb-2423-4182-9a77-b0d7c284f208\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6d9d59446f-spptn"
Jul 13 19:29:31.155003 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:29:31.154958    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f5a7e70-d1ae-4938-bd11-e286e8bac45a-dshm\") pod \"custom-route-timeout-pd-test-kserve-prefill-f7f58c4f6-nsctq\" (UID: \"6f5a7e70-d1ae-4938-bd11-e286e8bac45a\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-f7f58c4f6-nsctq"
Jul 13 19:29:31.155003 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:29:31.154998    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f5a7e70-d1ae-4938-bd11-e286e8bac45a-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-prefill-f7f58c4f6-nsctq\" (UID: \"6f5a7e70-d1ae-4938-bd11-e286e8bac45a\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-f7f58c4f6-nsctq"
Jul 13 19:29:31.155355 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:29:31.155036    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/62328ddb-2423-4182-9a77-b0d7c284f208-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-6d9d59446f-spptn\" (UID: \"62328ddb-2423-4182-9a77-b0d7c284f208\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6d9d59446f-spptn"
Jul 13 19:29:31.155355 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:29:31.155048    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62328ddb-2423-4182-9a77-b0d7c284f208-home\") pod \"custom-route-timeout-pd-test-kserve-6d9d59446f-spptn\" (UID: \"62328ddb-2423-4182-9a77-b0d7c284f208\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6d9d59446f-spptn"
Jul 13 19:29:31.155355 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:29:31.155137    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6f5a7e70-d1ae-4938-bd11-e286e8bac45a-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-prefill-f7f58c4f6-nsctq\" (UID: \"6f5a7e70-d1ae-4938-bd11-e286e8bac45a\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-f7f58c4f6-nsctq"
Jul 13 19:29:31.155355 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:29:31.155149    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f5a7e70-d1ae-4938-bd11-e286e8bac45a-home\") pod \"custom-route-timeout-pd-test-kserve-prefill-f7f58c4f6-nsctq\" (UID: \"6f5a7e70-d1ae-4938-bd11-e286e8bac45a\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-f7f58c4f6-nsctq"
Jul 13 19:29:31.155355 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:29:31.155289    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f5a7e70-d1ae-4938-bd11-e286e8bac45a-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-prefill-f7f58c4f6-nsctq\" (UID: \"6f5a7e70-d1ae-4938-bd11-e286e8bac45a\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-f7f58c4f6-nsctq"
Jul 13 19:29:31.155806 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:29:31.155549    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/62328ddb-2423-4182-9a77-b0d7c284f208-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-6d9d59446f-spptn\" (UID: \"62328ddb-2423-4182-9a77-b0d7c284f208\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6d9d59446f-spptn"
Jul 13 19:29:31.155806 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:29:31.155641    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62328ddb-2423-4182-9a77-b0d7c284f208-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-6d9d59446f-spptn\" (UID: \"62328ddb-2423-4182-9a77-b0d7c284f208\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6d9d59446f-spptn"
Jul 13 19:29:31.155927 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:29:31.155815    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62328ddb-2423-4182-9a77-b0d7c284f208-model-cache\") pod \"custom-route-timeout-pd-test-kserve-6d9d59446f-spptn\" (UID: \"62328ddb-2423-4182-9a77-b0d7c284f208\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6d9d59446f-spptn"
Jul 13 19:29:31.155927 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:29:31.155844    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f5a7e70-d1ae-4938-bd11-e286e8bac45a-model-cache\") pod \"custom-route-timeout-pd-test-kserve-prefill-f7f58c4f6-nsctq\" (UID: \"6f5a7e70-d1ae-4938-bd11-e286e8bac45a\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-f7f58c4f6-nsctq"
Jul 13 19:29:31.157406 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:29:31.157383    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f5a7e70-d1ae-4938-bd11-e286e8bac45a-dshm\") pod \"custom-route-timeout-pd-test-kserve-prefill-f7f58c4f6-nsctq\" (UID: \"6f5a7e70-d1ae-4938-bd11-e286e8bac45a\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-f7f58c4f6-nsctq"
Jul 13 19:29:31.157622 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:29:31.157599    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f5a7e70-d1ae-4938-bd11-e286e8bac45a-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-prefill-f7f58c4f6-nsctq\" (UID: \"6f5a7e70-d1ae-4938-bd11-e286e8bac45a\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-f7f58c4f6-nsctq"
Jul 13 19:29:31.157669 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:29:31.157633    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62328ddb-2423-4182-9a77-b0d7c284f208-dshm\") pod \"custom-route-timeout-pd-test-kserve-6d9d59446f-spptn\" (UID: \"62328ddb-2423-4182-9a77-b0d7c284f208\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6d9d59446f-spptn"
Jul 13 19:29:31.157942 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:29:31.157927    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62328ddb-2423-4182-9a77-b0d7c284f208-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-6d9d59446f-spptn\" (UID: \"62328ddb-2423-4182-9a77-b0d7c284f208\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6d9d59446f-spptn"
Jul 13 19:29:31.169537 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:29:31.169514    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxwl6\" (UniqueName: \"kubernetes.io/projected/6f5a7e70-d1ae-4938-bd11-e286e8bac45a-kube-api-access-dxwl6\") pod \"custom-route-timeout-pd-test-kserve-prefill-f7f58c4f6-nsctq\" (UID: \"6f5a7e70-d1ae-4938-bd11-e286e8bac45a\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-f7f58c4f6-nsctq"
Jul 13 19:29:31.170504 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:29:31.170479    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzq2p\" (UniqueName: \"kubernetes.io/projected/62328ddb-2423-4182-9a77-b0d7c284f208-kube-api-access-tzq2p\") pod \"custom-route-timeout-pd-test-kserve-6d9d59446f-spptn\" (UID: \"62328ddb-2423-4182-9a77-b0d7c284f208\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6d9d59446f-spptn"
Jul 13 19:29:31.289116 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:29:31.289086    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6d9d59446f-spptn"
Jul 13 19:29:31.298866 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:29:31.298842    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-f7f58c4f6-nsctq"
Jul 13 19:29:31.448963 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:29:31.448935    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6d9d59446f-spptn"]
Jul 13 19:29:31.450684 ip-10-0-128-215 kubenswrapper[2565]: W0713 19:29:31.450651    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62328ddb_2423_4182_9a77_b0d7c284f208.slice/crio-bd054b225286b7662439d778077e06ad971a86b0577cd55bca04b4b2ceea0b29 WatchSource:0}: Error finding container bd054b225286b7662439d778077e06ad971a86b0577cd55bca04b4b2ceea0b29: Status 404 returned error can't find the container with id bd054b225286b7662439d778077e06ad971a86b0577cd55bca04b4b2ceea0b29
Jul 13 19:29:31.473966 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:29:31.471473    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-f7f58c4f6-nsctq"]
Jul 13 19:29:32.315308 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:29:32.315276    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6d9d59446f-spptn" event={"ID":"62328ddb-2423-4182-9a77-b0d7c284f208","Type":"ContainerStarted","Data":"dd69052893104fd85d1e7af38e4347d4e031db61e57c4c643cd85da94391855c"}
Jul 13 19:29:32.315759 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:29:32.315331    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6d9d59446f-spptn" event={"ID":"62328ddb-2423-4182-9a77-b0d7c284f208","Type":"ContainerStarted","Data":"bd054b225286b7662439d778077e06ad971a86b0577cd55bca04b4b2ceea0b29"}
Jul 13 19:29:32.315759 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:29:32.315359    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6d9d59446f-spptn"
Jul 13 19:29:32.316900 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:29:32.316866    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-f7f58c4f6-nsctq" event={"ID":"6f5a7e70-d1ae-4938-bd11-e286e8bac45a","Type":"ContainerStarted","Data":"c36b62c8ecbee3734e7f5595093b9bd86100782ca411fa6312d2e9ae3a4a77bb"}
Jul 13 19:29:32.317011 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:29:32.316914    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-f7f58c4f6-nsctq" event={"ID":"6f5a7e70-d1ae-4938-bd11-e286e8bac45a","Type":"ContainerStarted","Data":"5c0fc96cfec5c383e55eec69f5dc7b5c9a3c9d5d7d55775605edc8ab814f2007"}
Jul 13 19:29:33.326454 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:29:33.326406    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6d9d59446f-spptn" event={"ID":"62328ddb-2423-4182-9a77-b0d7c284f208","Type":"ContainerStarted","Data":"a7f2804e3078bece3cbf0995431cb8f53432694d6aff1bdbe8a8d858f69ebc62"}
Jul 13 19:29:35.343647 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:29:35.343608    2565 generic.go:356] "Generic (PLEG): container finished" podID="6f5a7e70-d1ae-4938-bd11-e286e8bac45a" containerID="c36b62c8ecbee3734e7f5595093b9bd86100782ca411fa6312d2e9ae3a4a77bb" exitCode=0
Jul 13 19:29:35.344387 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:29:35.343692    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-f7f58c4f6-nsctq" event={"ID":"6f5a7e70-d1ae-4938-bd11-e286e8bac45a","Type":"ContainerDied","Data":"c36b62c8ecbee3734e7f5595093b9bd86100782ca411fa6312d2e9ae3a4a77bb"}
Jul 13 19:29:36.358995 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:29:36.358955    2565 generic.go:356] "Generic (PLEG): container finished" podID="62328ddb-2423-4182-9a77-b0d7c284f208" containerID="a7f2804e3078bece3cbf0995431cb8f53432694d6aff1bdbe8a8d858f69ebc62" exitCode=0
Jul 13 19:29:36.359631 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:29:36.358991    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6d9d59446f-spptn" event={"ID":"62328ddb-2423-4182-9a77-b0d7c284f208","Type":"ContainerDied","Data":"a7f2804e3078bece3cbf0995431cb8f53432694d6aff1bdbe8a8d858f69ebc62"}
Jul 13 19:29:36.361130 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:29:36.361052    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-f7f58c4f6-nsctq" event={"ID":"6f5a7e70-d1ae-4938-bd11-e286e8bac45a","Type":"ContainerStarted","Data":"7c3d8d7fd7089da663fec37b8efa8167fd5cdc850968b33afe4c58bd656e52ef"}
Jul 13 19:29:36.410217 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:29:36.410164    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-f7f58c4f6-nsctq" podStartSLOduration=3.352804956 podStartE2EDuration="6.410142778s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.057337822s" isStatefulPod=false podCreationTimestamp="2026-07-13 19:29:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 19:29:36.408898868 +0000 UTC m=+2362.934032309" watchObservedRunningTime="2026-07-13 19:29:36.410142778 +0000 UTC m=+2362.935276195"
Jul 13 19:29:37.369519 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:29:37.369480    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6d9d59446f-spptn" event={"ID":"62328ddb-2423-4182-9a77-b0d7c284f208","Type":"ContainerStarted","Data":"3fe3def0e2e0709639ec18ba35991a379c0613e62385db4f37014547b28b9a12"}
Jul 13 19:29:37.400191 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:29:37.400135    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6d9d59446f-spptn" podStartSLOduration=4.158031595 podStartE2EDuration="7.400119006s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.242087411s" isStatefulPod=false podCreationTimestamp="2026-07-13 19:29:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 19:29:37.396881909 +0000 UTC m=+2363.922015326" watchObservedRunningTime="2026-07-13 19:29:37.400119006 +0000 UTC m=+2363.925252422"
Jul 13 19:29:41.290054 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:29:41.290012    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6d9d59446f-spptn"
Jul 13 19:29:41.290571 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:29:41.290094    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6d9d59446f-spptn"
Jul 13 19:29:41.291693 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:29:41.291658    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6d9d59446f-spptn" podUID="62328ddb-2423-4182-9a77-b0d7c284f208" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8001/health\": dial tcp 10.134.0.55:8001: connect: connection refused"
Jul 13 19:29:41.299276 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:29:41.299249    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-f7f58c4f6-nsctq"
Jul 13 19:29:41.299406 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:29:41.299312    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-f7f58c4f6-nsctq"
Jul 13 19:29:41.300484 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:29:41.300453    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-f7f58c4f6-nsctq" podUID="6f5a7e70-d1ae-4938-bd11-e286e8bac45a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": dial tcp 10.134.0.56:8000: connect: connection refused"
Jul 13 19:29:41.304055 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:29:41.304034    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6d9d59446f-spptn"
Jul 13 19:29:51.290269 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:29:51.290226    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6d9d59446f-spptn" podUID="62328ddb-2423-4182-9a77-b0d7c284f208" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8001/health\": dial tcp 10.134.0.55:8001: connect: connection refused"
Jul 13 19:29:51.300526 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:29:51.300444    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-f7f58c4f6-nsctq" podUID="6f5a7e70-d1ae-4938-bd11-e286e8bac45a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": dial tcp 10.134.0.56:8000: connect: connection refused"
Jul 13 19:30:01.289968 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:30:01.289908    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6d9d59446f-spptn" podUID="62328ddb-2423-4182-9a77-b0d7c284f208" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8001/health\": dial tcp 10.134.0.55:8001: connect: connection refused"
Jul 13 19:30:01.299958 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:30:01.299907    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-f7f58c4f6-nsctq" podUID="6f5a7e70-d1ae-4938-bd11-e286e8bac45a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": dial tcp 10.134.0.56:8000: connect: connection refused"
Jul 13 19:30:11.289655 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:30:11.289602    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6d9d59446f-spptn" podUID="62328ddb-2423-4182-9a77-b0d7c284f208" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8001/health\": dial tcp 10.134.0.55:8001: connect: connection refused"
Jul 13 19:30:11.299441 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:30:11.299401    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-f7f58c4f6-nsctq" podUID="6f5a7e70-d1ae-4938-bd11-e286e8bac45a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": dial tcp 10.134.0.56:8000: connect: connection refused"
Jul 13 19:30:12.886238 ip-10-0-128-215 kubenswrapper[2565]: E0713 19:30:12.886192    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dmww7f" podUID="634b78c4-751c-4880-9f6c-1ea69b737987"
Jul 13 19:30:13.589646 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:30:13.589607    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dmww7f"
Jul 13 19:30:14.113767 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:30:14.113737    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2bscc_2f0655bd-4a09-4314-8c1b-e30191eaec0e/ovn-acl-logging/0.log"
Jul 13 19:30:14.115372 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:30:14.115346    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2bscc_2f0655bd-4a09-4314-8c1b-e30191eaec0e/ovn-acl-logging/0.log"
Jul 13 19:30:21.290130 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:30:21.290083    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6d9d59446f-spptn" podUID="62328ddb-2423-4182-9a77-b0d7c284f208" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8001/health\": dial tcp 10.134.0.55:8001: connect: connection refused"
Jul 13 19:30:21.300138 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:30:21.300090    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-f7f58c4f6-nsctq" podUID="6f5a7e70-d1ae-4938-bd11-e286e8bac45a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": dial tcp 10.134.0.56:8000: connect: connection refused"
Jul 13 19:30:31.290528 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:30:31.290477    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6d9d59446f-spptn" podUID="62328ddb-2423-4182-9a77-b0d7c284f208" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8001/health\": dial tcp 10.134.0.55:8001: connect: connection refused"
Jul 13 19:30:31.299405 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:30:31.299371    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-f7f58c4f6-nsctq" podUID="6f5a7e70-d1ae-4938-bd11-e286e8bac45a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": dial tcp 10.134.0.56:8000: connect: connection refused"
Jul 13 19:30:41.289908 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:30:41.289811    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6d9d59446f-spptn" podUID="62328ddb-2423-4182-9a77-b0d7c284f208" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8001/health\": dial tcp 10.134.0.55:8001: connect: connection refused"
Jul 13 19:30:41.299511 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:30:41.299474    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-f7f58c4f6-nsctq" podUID="6f5a7e70-d1ae-4938-bd11-e286e8bac45a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": dial tcp 10.134.0.56:8000: connect: connection refused"
Jul 13 19:30:51.290007 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:30:51.289956    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6d9d59446f-spptn" podUID="62328ddb-2423-4182-9a77-b0d7c284f208" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8001/health\": dial tcp 10.134.0.55:8001: connect: connection refused"
Jul 13 19:30:51.299904 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:30:51.299871    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-f7f58c4f6-nsctq" podUID="6f5a7e70-d1ae-4938-bd11-e286e8bac45a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": dial tcp 10.134.0.56:8000: connect: connection refused"
Jul 13 19:31:01.290583 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:31:01.290532    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6d9d59446f-spptn" podUID="62328ddb-2423-4182-9a77-b0d7c284f208" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8001/health\": dial tcp 10.134.0.55:8001: connect: connection refused"
Jul 13 19:31:01.299178 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:31:01.299138    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-f7f58c4f6-nsctq" podUID="6f5a7e70-d1ae-4938-bd11-e286e8bac45a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": dial tcp 10.134.0.56:8000: connect: connection refused"
Jul 13 19:31:11.290268 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:31:11.290210    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6d9d59446f-spptn" podUID="62328ddb-2423-4182-9a77-b0d7c284f208" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8001/health\": dial tcp 10.134.0.55:8001: connect: connection refused"
Jul 13 19:31:11.299483 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:31:11.299453    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-f7f58c4f6-nsctq" podUID="6f5a7e70-d1ae-4938-bd11-e286e8bac45a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": dial tcp 10.134.0.56:8000: connect: connection refused"
Jul 13 19:31:21.289921 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:31:21.289864    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6d9d59446f-spptn" podUID="62328ddb-2423-4182-9a77-b0d7c284f208" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8001/health\": dial tcp 10.134.0.55:8001: connect: connection refused"
Jul 13 19:31:21.299447 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:31:21.299416    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-f7f58c4f6-nsctq" podUID="6f5a7e70-d1ae-4938-bd11-e286e8bac45a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": dial tcp 10.134.0.56:8000: connect: connection refused"
Jul 13 19:31:31.290448 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:31:31.290398    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6d9d59446f-spptn" podUID="62328ddb-2423-4182-9a77-b0d7c284f208" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8001/health\": dial tcp 10.134.0.55:8001: connect: connection refused"
Jul 13 19:31:31.299526 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:31:31.299498    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-f7f58c4f6-nsctq" podUID="6f5a7e70-d1ae-4938-bd11-e286e8bac45a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": dial tcp 10.134.0.56:8000: connect: connection refused"
Jul 13 19:31:41.289881 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:31:41.289832    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6d9d59446f-spptn" podUID="62328ddb-2423-4182-9a77-b0d7c284f208" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8001/health\": dial tcp 10.134.0.55:8001: connect: connection refused"
Jul 13 19:31:41.300058 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:31:41.300016    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-f7f58c4f6-nsctq" podUID="6f5a7e70-d1ae-4938-bd11-e286e8bac45a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": dial tcp 10.134.0.56:8000: connect: connection refused"
Jul 13 19:31:51.289684 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:31:51.289636    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6d9d59446f-spptn" podUID="62328ddb-2423-4182-9a77-b0d7c284f208" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8001/health\": dial tcp 10.134.0.55:8001: connect: connection refused"
Jul 13 19:31:51.299486 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:31:51.299454    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-f7f58c4f6-nsctq" podUID="6f5a7e70-d1ae-4938-bd11-e286e8bac45a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": dial tcp 10.134.0.56:8000: connect: connection refused"
Jul 13 19:32:01.290424 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:32:01.290373    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6d9d59446f-spptn" podUID="62328ddb-2423-4182-9a77-b0d7c284f208" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8001/health\": dial tcp 10.134.0.55:8001: connect: connection refused"
Jul 13 19:32:01.299627 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:32:01.299593    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-f7f58c4f6-nsctq" podUID="6f5a7e70-d1ae-4938-bd11-e286e8bac45a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": dial tcp 10.134.0.56:8000: connect: connection refused"
Jul 13 19:32:11.299568 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:32:11.299478    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6d9d59446f-spptn"
Jul 13 19:32:11.310004 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:32:11.309977    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-f7f58c4f6-nsctq"
Jul 13 19:32:11.311498 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:32:11.311475    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6d9d59446f-spptn"
Jul 13 19:32:11.318192 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:32:11.318170    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-f7f58c4f6-nsctq"
Jul 13 19:32:16.591087 ip-10-0-128-215 kubenswrapper[2565]: E0713 19:32:16.591026    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dmww7f" podUID="634b78c4-751c-4880-9f6c-1ea69b737987"
Jul 13 19:32:17.281767 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:32:17.281733    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dmww7f"
Jul 13 19:32:59.550398 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:32:59.550356    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-f7f58c4f6-nsctq"]
Jul 13 19:32:59.555523 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:32:59.555496    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6d9d59446f-spptn"]
Jul 13 19:33:14.575904 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:33:14.575846    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-f7f58c4f6-nsctq" podUID="6f5a7e70-d1ae-4938-bd11-e286e8bac45a" containerName="main" containerID="cri-o://7c3d8d7fd7089da663fec37b8efa8167fd5cdc850968b33afe4c58bd656e52ef" gracePeriod=45
Jul 13 19:33:14.580837 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:33:14.580798    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6d9d59446f-spptn" podUID="62328ddb-2423-4182-9a77-b0d7c284f208" containerName="main" containerID="cri-o://3fe3def0e2e0709639ec18ba35991a379c0613e62385db4f37014547b28b9a12" gracePeriod=45
Jul 13 19:33:15.335137 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:33:15.335100    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6d9d59446f-spptn" podUID="62328ddb-2423-4182-9a77-b0d7c284f208" containerName="llm-d-routing-sidecar" containerID="cri-o://dd69052893104fd85d1e7af38e4347d4e031db61e57c4c643cd85da94391855c" gracePeriod=45
Jul 13 19:33:15.461198 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:33:15.461174    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-f7f58c4f6-nsctq"
Jul 13 19:33:15.538313 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:33:15.538249    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dxwl6\" (UniqueName: \"kubernetes.io/projected/6f5a7e70-d1ae-4938-bd11-e286e8bac45a-kube-api-access-dxwl6\") pod \"6f5a7e70-d1ae-4938-bd11-e286e8bac45a\" (UID: \"6f5a7e70-d1ae-4938-bd11-e286e8bac45a\") "
Jul 13 19:33:15.538313 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:33:15.538287    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f5a7e70-d1ae-4938-bd11-e286e8bac45a-model-cache\") pod \"6f5a7e70-d1ae-4938-bd11-e286e8bac45a\" (UID: \"6f5a7e70-d1ae-4938-bd11-e286e8bac45a\") "
Jul 13 19:33:15.538528 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:33:15.538317    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f5a7e70-d1ae-4938-bd11-e286e8bac45a-dshm\") pod \"6f5a7e70-d1ae-4938-bd11-e286e8bac45a\" (UID: \"6f5a7e70-d1ae-4938-bd11-e286e8bac45a\") "
Jul 13 19:33:15.538528 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:33:15.538354    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6f5a7e70-d1ae-4938-bd11-e286e8bac45a-kserve-provision-location\") pod \"6f5a7e70-d1ae-4938-bd11-e286e8bac45a\" (UID: \"6f5a7e70-d1ae-4938-bd11-e286e8bac45a\") "
Jul 13 19:33:15.538528 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:33:15.538385    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f5a7e70-d1ae-4938-bd11-e286e8bac45a-tls-certs\") pod \"6f5a7e70-d1ae-4938-bd11-e286e8bac45a\" (UID: \"6f5a7e70-d1ae-4938-bd11-e286e8bac45a\") "
Jul 13 19:33:15.538528 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:33:15.538457    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f5a7e70-d1ae-4938-bd11-e286e8bac45a-tmp-dir\") pod \"6f5a7e70-d1ae-4938-bd11-e286e8bac45a\" (UID: \"6f5a7e70-d1ae-4938-bd11-e286e8bac45a\") "
Jul 13 19:33:15.538528 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:33:15.538486    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f5a7e70-d1ae-4938-bd11-e286e8bac45a-home\") pod \"6f5a7e70-d1ae-4938-bd11-e286e8bac45a\" (UID: \"6f5a7e70-d1ae-4938-bd11-e286e8bac45a\") "
Jul 13 19:33:15.538828 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:33:15.538563    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f5a7e70-d1ae-4938-bd11-e286e8bac45a-model-cache" (OuterVolumeSpecName: "model-cache") pod "6f5a7e70-d1ae-4938-bd11-e286e8bac45a" (UID: "6f5a7e70-d1ae-4938-bd11-e286e8bac45a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:33:15.538889 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:33:15.538859    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f5a7e70-d1ae-4938-bd11-e286e8bac45a-model-cache\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 13 19:33:15.539453 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:33:15.539427    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f5a7e70-d1ae-4938-bd11-e286e8bac45a-home" (OuterVolumeSpecName: "home") pod "6f5a7e70-d1ae-4938-bd11-e286e8bac45a" (UID: "6f5a7e70-d1ae-4938-bd11-e286e8bac45a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:33:15.540678 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:33:15.540643    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f5a7e70-d1ae-4938-bd11-e286e8bac45a-dshm" (OuterVolumeSpecName: "dshm") pod "6f5a7e70-d1ae-4938-bd11-e286e8bac45a" (UID: "6f5a7e70-d1ae-4938-bd11-e286e8bac45a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:33:15.540803 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:33:15.540738    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f5a7e70-d1ae-4938-bd11-e286e8bac45a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6f5a7e70-d1ae-4938-bd11-e286e8bac45a" (UID: "6f5a7e70-d1ae-4938-bd11-e286e8bac45a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 19:33:15.541131 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:33:15.541101    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f5a7e70-d1ae-4938-bd11-e286e8bac45a-kube-api-access-dxwl6" (OuterVolumeSpecName: "kube-api-access-dxwl6") pod "6f5a7e70-d1ae-4938-bd11-e286e8bac45a" (UID: "6f5a7e70-d1ae-4938-bd11-e286e8bac45a"). InnerVolumeSpecName "kube-api-access-dxwl6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 19:33:15.557919 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:33:15.557886    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f5a7e70-d1ae-4938-bd11-e286e8bac45a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6f5a7e70-d1ae-4938-bd11-e286e8bac45a" (UID: "6f5a7e70-d1ae-4938-bd11-e286e8bac45a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:33:15.576300 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:33:15.576272    2565 generic.go:356] "Generic (PLEG): container finished" podID="62328ddb-2423-4182-9a77-b0d7c284f208" containerID="3fe3def0e2e0709639ec18ba35991a379c0613e62385db4f37014547b28b9a12" exitCode=0
Jul 13 19:33:15.576300 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:33:15.576298    2565 generic.go:356] "Generic (PLEG): container finished" podID="62328ddb-2423-4182-9a77-b0d7c284f208" containerID="dd69052893104fd85d1e7af38e4347d4e031db61e57c4c643cd85da94391855c" exitCode=0
Jul 13 19:33:15.576786 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:33:15.576347    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6d9d59446f-spptn" event={"ID":"62328ddb-2423-4182-9a77-b0d7c284f208","Type":"ContainerDied","Data":"3fe3def0e2e0709639ec18ba35991a379c0613e62385db4f37014547b28b9a12"}
Jul 13 19:33:15.576786 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:33:15.576418    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6d9d59446f-spptn" event={"ID":"62328ddb-2423-4182-9a77-b0d7c284f208","Type":"ContainerDied","Data":"dd69052893104fd85d1e7af38e4347d4e031db61e57c4c643cd85da94391855c"}
Jul 13 19:33:15.578374 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:33:15.578343    2565 generic.go:356] "Generic (PLEG): container finished" podID="6f5a7e70-d1ae-4938-bd11-e286e8bac45a" containerID="7c3d8d7fd7089da663fec37b8efa8167fd5cdc850968b33afe4c58bd656e52ef" exitCode=0
Jul 13 19:33:15.578551 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:33:15.578398    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-f7f58c4f6-nsctq"
Jul 13 19:33:15.578551 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:33:15.578425    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-f7f58c4f6-nsctq" event={"ID":"6f5a7e70-d1ae-4938-bd11-e286e8bac45a","Type":"ContainerDied","Data":"7c3d8d7fd7089da663fec37b8efa8167fd5cdc850968b33afe4c58bd656e52ef"}
Jul 13 19:33:15.578551 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:33:15.578483    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-f7f58c4f6-nsctq" event={"ID":"6f5a7e70-d1ae-4938-bd11-e286e8bac45a","Type":"ContainerDied","Data":"5c0fc96cfec5c383e55eec69f5dc7b5c9a3c9d5d7d55775605edc8ab814f2007"}
Jul 13 19:33:15.578551 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:33:15.578509    2565 scope.go:117] "RemoveContainer" containerID="7c3d8d7fd7089da663fec37b8efa8167fd5cdc850968b33afe4c58bd656e52ef"
Jul 13 19:33:15.590572 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:33:15.589848    2565 scope.go:117] "RemoveContainer" containerID="c36b62c8ecbee3734e7f5595093b9bd86100782ca411fa6312d2e9ae3a4a77bb"
Jul 13 19:33:15.590657 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:33:15.590618    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6d9d59446f-spptn"
Jul 13 19:33:15.599910 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:33:15.599895    2565 scope.go:117] "RemoveContainer" containerID="7c3d8d7fd7089da663fec37b8efa8167fd5cdc850968b33afe4c58bd656e52ef"
Jul 13 19:33:15.600168 ip-10-0-128-215 kubenswrapper[2565]: E0713 19:33:15.600151    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c3d8d7fd7089da663fec37b8efa8167fd5cdc850968b33afe4c58bd656e52ef\": container with ID starting with 7c3d8d7fd7089da663fec37b8efa8167fd5cdc850968b33afe4c58bd656e52ef not found: ID does not exist" containerID="7c3d8d7fd7089da663fec37b8efa8167fd5cdc850968b33afe4c58bd656e52ef"
Jul 13 19:33:15.600223 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:33:15.600177    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c3d8d7fd7089da663fec37b8efa8167fd5cdc850968b33afe4c58bd656e52ef"} err="failed to get container status \"7c3d8d7fd7089da663fec37b8efa8167fd5cdc850968b33afe4c58bd656e52ef\": rpc error: code = NotFound desc = could not find container \"7c3d8d7fd7089da663fec37b8efa8167fd5cdc850968b33afe4c58bd656e52ef\": container with ID starting with 7c3d8d7fd7089da663fec37b8efa8167fd5cdc850968b33afe4c58bd656e52ef not found: ID does not exist"
Jul 13 19:33:15.600223 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:33:15.600194    2565 scope.go:117] "RemoveContainer" containerID="c36b62c8ecbee3734e7f5595093b9bd86100782ca411fa6312d2e9ae3a4a77bb"
Jul 13 19:33:15.600431 ip-10-0-128-215 kubenswrapper[2565]: E0713 19:33:15.600412    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c36b62c8ecbee3734e7f5595093b9bd86100782ca411fa6312d2e9ae3a4a77bb\": container with ID starting with c36b62c8ecbee3734e7f5595093b9bd86100782ca411fa6312d2e9ae3a4a77bb not found: ID does not exist" containerID="c36b62c8ecbee3734e7f5595093b9bd86100782ca411fa6312d2e9ae3a4a77bb"
Jul 13 19:33:15.600476 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:33:15.600437    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c36b62c8ecbee3734e7f5595093b9bd86100782ca411fa6312d2e9ae3a4a77bb"} err="failed to get container status \"c36b62c8ecbee3734e7f5595093b9bd86100782ca411fa6312d2e9ae3a4a77bb\": rpc error: code = NotFound desc = could not find container \"c36b62c8ecbee3734e7f5595093b9bd86100782ca411fa6312d2e9ae3a4a77bb\": container with ID starting with c36b62c8ecbee3734e7f5595093b9bd86100782ca411fa6312d2e9ae3a4a77bb not found: ID does not exist"
Jul 13 19:33:15.621922 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:33:15.621890    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f5a7e70-d1ae-4938-bd11-e286e8bac45a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6f5a7e70-d1ae-4938-bd11-e286e8bac45a" (UID: "6f5a7e70-d1ae-4938-bd11-e286e8bac45a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:33:15.639573 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:33:15.639539    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tzq2p\" (UniqueName: \"kubernetes.io/projected/62328ddb-2423-4182-9a77-b0d7c284f208-kube-api-access-tzq2p\") pod \"62328ddb-2423-4182-9a77-b0d7c284f208\" (UID: \"62328ddb-2423-4182-9a77-b0d7c284f208\") "
Jul 13 19:33:15.639762 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:33:15.639622    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/62328ddb-2423-4182-9a77-b0d7c284f208-kserve-provision-location\") pod \"62328ddb-2423-4182-9a77-b0d7c284f208\" (UID: \"62328ddb-2423-4182-9a77-b0d7c284f208\") "
Jul 13 19:33:15.639762 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:33:15.639653    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62328ddb-2423-4182-9a77-b0d7c284f208-tls-certs\") pod \"62328ddb-2423-4182-9a77-b0d7c284f208\" (UID: \"62328ddb-2423-4182-9a77-b0d7c284f208\") "
Jul 13 19:33:15.639762 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:33:15.639679    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62328ddb-2423-4182-9a77-b0d7c284f208-tmp-dir\") pod \"62328ddb-2423-4182-9a77-b0d7c284f208\" (UID: \"62328ddb-2423-4182-9a77-b0d7c284f208\") "
Jul 13 19:33:15.639762 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:33:15.639759    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62328ddb-2423-4182-9a77-b0d7c284f208-home\") pod \"62328ddb-2423-4182-9a77-b0d7c284f208\" (UID: \"62328ddb-2423-4182-9a77-b0d7c284f208\") "
Jul 13 19:33:15.639991 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:33:15.639795    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62328ddb-2423-4182-9a77-b0d7c284f208-dshm\") pod \"62328ddb-2423-4182-9a77-b0d7c284f208\" (UID: \"62328ddb-2423-4182-9a77-b0d7c284f208\") "
Jul 13 19:33:15.639991 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:33:15.639882    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62328ddb-2423-4182-9a77-b0d7c284f208-model-cache\") pod \"62328ddb-2423-4182-9a77-b0d7c284f208\" (UID: \"62328ddb-2423-4182-9a77-b0d7c284f208\") "
Jul 13 19:33:15.640214 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:33:15.640195    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f5a7e70-d1ae-4938-bd11-e286e8bac45a-tmp-dir\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 13 19:33:15.640294 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:33:15.640222    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f5a7e70-d1ae-4938-bd11-e286e8bac45a-home\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 13 19:33:15.640294 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:33:15.640236    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dxwl6\" (UniqueName: \"kubernetes.io/projected/6f5a7e70-d1ae-4938-bd11-e286e8bac45a-kube-api-access-dxwl6\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 13 19:33:15.640294 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:33:15.640249    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f5a7e70-d1ae-4938-bd11-e286e8bac45a-dshm\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 13 19:33:15.640294 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:33:15.640264    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6f5a7e70-d1ae-4938-bd11-e286e8bac45a-kserve-provision-location\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 13 19:33:15.640294 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:33:15.640278    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f5a7e70-d1ae-4938-bd11-e286e8bac45a-tls-certs\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 13 19:33:15.640734 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:33:15.640379    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62328ddb-2423-4182-9a77-b0d7c284f208-model-cache" (OuterVolumeSpecName: "model-cache") pod "62328ddb-2423-4182-9a77-b0d7c284f208" (UID: "62328ddb-2423-4182-9a77-b0d7c284f208"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:33:15.640734 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:33:15.640478    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62328ddb-2423-4182-9a77-b0d7c284f208-home" (OuterVolumeSpecName: "home") pod "62328ddb-2423-4182-9a77-b0d7c284f208" (UID: "62328ddb-2423-4182-9a77-b0d7c284f208"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:33:15.642178 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:33:15.642147    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62328ddb-2423-4182-9a77-b0d7c284f208-kube-api-access-tzq2p" (OuterVolumeSpecName: "kube-api-access-tzq2p") pod "62328ddb-2423-4182-9a77-b0d7c284f208" (UID: "62328ddb-2423-4182-9a77-b0d7c284f208"). InnerVolumeSpecName "kube-api-access-tzq2p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 19:33:15.642359 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:33:15.642338    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62328ddb-2423-4182-9a77-b0d7c284f208-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "62328ddb-2423-4182-9a77-b0d7c284f208" (UID: "62328ddb-2423-4182-9a77-b0d7c284f208"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 19:33:15.642653 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:33:15.642623    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62328ddb-2423-4182-9a77-b0d7c284f208-dshm" (OuterVolumeSpecName: "dshm") pod "62328ddb-2423-4182-9a77-b0d7c284f208" (UID: "62328ddb-2423-4182-9a77-b0d7c284f208"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:33:15.654319 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:33:15.654284    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62328ddb-2423-4182-9a77-b0d7c284f208-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "62328ddb-2423-4182-9a77-b0d7c284f208" (UID: "62328ddb-2423-4182-9a77-b0d7c284f208"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:33:15.704448 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:33:15.704407    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62328ddb-2423-4182-9a77-b0d7c284f208-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "62328ddb-2423-4182-9a77-b0d7c284f208" (UID: "62328ddb-2423-4182-9a77-b0d7c284f208"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:33:15.741319 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:33:15.741283    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62328ddb-2423-4182-9a77-b0d7c284f208-model-cache\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 13 19:33:15.741319 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:33:15.741307    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tzq2p\" (UniqueName: \"kubernetes.io/projected/62328ddb-2423-4182-9a77-b0d7c284f208-kube-api-access-tzq2p\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 13 19:33:15.741319 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:33:15.741317    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/62328ddb-2423-4182-9a77-b0d7c284f208-kserve-provision-location\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 13 19:33:15.741523 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:33:15.741326    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62328ddb-2423-4182-9a77-b0d7c284f208-tls-certs\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 13 19:33:15.741523 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:33:15.741336    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62328ddb-2423-4182-9a77-b0d7c284f208-tmp-dir\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 13 19:33:15.741523 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:33:15.741343    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62328ddb-2423-4182-9a77-b0d7c284f208-home\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 13 19:33:15.741523 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:33:15.741350    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62328ddb-2423-4182-9a77-b0d7c284f208-dshm\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 13 19:33:15.901848 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:33:15.901819    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-f7f58c4f6-nsctq"]
Jul 13 19:33:15.906273 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:33:15.906240    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-f7f58c4f6-nsctq"]
Jul 13 19:33:16.051162 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:33:16.051124    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6f5a7e70-d1ae-4938-bd11-e286e8bac45a" path="/var/lib/kubelet/pods/6f5a7e70-d1ae-4938-bd11-e286e8bac45a/volumes"
Jul 13 19:33:16.585468 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:33:16.585383    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6d9d59446f-spptn"
Jul 13 19:33:16.585468 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:33:16.585397    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6d9d59446f-spptn" event={"ID":"62328ddb-2423-4182-9a77-b0d7c284f208","Type":"ContainerDied","Data":"bd054b225286b7662439d778077e06ad971a86b0577cd55bca04b4b2ceea0b29"}
Jul 13 19:33:16.585468 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:33:16.585464    2565 scope.go:117] "RemoveContainer" containerID="3fe3def0e2e0709639ec18ba35991a379c0613e62385db4f37014547b28b9a12"
Jul 13 19:33:16.594367 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:33:16.594350    2565 scope.go:117] "RemoveContainer" containerID="a7f2804e3078bece3cbf0995431cb8f53432694d6aff1bdbe8a8d858f69ebc62"
Jul 13 19:33:16.603047 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:33:16.603028    2565 scope.go:117] "RemoveContainer" containerID="dd69052893104fd85d1e7af38e4347d4e031db61e57c4c643cd85da94391855c"
Jul 13 19:33:16.606157 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:33:16.606138    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6d9d59446f-spptn"]
Jul 13 19:33:16.609963 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:33:16.609939    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6d9d59446f-spptn"]
Jul 13 19:33:18.048028 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:33:18.047993    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="62328ddb-2423-4182-9a77-b0d7c284f208" path="/var/lib/kubelet/pods/62328ddb-2423-4182-9a77-b0d7c284f208/volumes"
Jul 13 19:33:21.882860 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:33:21.882829    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mhsdz"]
Jul 13 19:33:21.883320 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:33:21.883216    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62328ddb-2423-4182-9a77-b0d7c284f208" containerName="storage-initializer"
Jul 13 19:33:21.883320 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:33:21.883227    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="62328ddb-2423-4182-9a77-b0d7c284f208" containerName="storage-initializer"
Jul 13 19:33:21.883320 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:33:21.883236    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62328ddb-2423-4182-9a77-b0d7c284f208" containerName="llm-d-routing-sidecar"
Jul 13 19:33:21.883320 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:33:21.883243    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="62328ddb-2423-4182-9a77-b0d7c284f208" containerName="llm-d-routing-sidecar"
Jul 13 19:33:21.883320 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:33:21.883249    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f5a7e70-d1ae-4938-bd11-e286e8bac45a" containerName="main"
Jul 13 19:33:21.883320 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:33:21.883255    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f5a7e70-d1ae-4938-bd11-e286e8bac45a" containerName="main"
Jul 13 19:33:21.883320 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:33:21.883266    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f5a7e70-d1ae-4938-bd11-e286e8bac45a" containerName="storage-initializer"
Jul 13 19:33:21.883320 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:33:21.883271    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f5a7e70-d1ae-4938-bd11-e286e8bac45a" containerName="storage-initializer"
Jul 13 19:33:21.883320 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:33:21.883277    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62328ddb-2423-4182-9a77-b0d7c284f208" containerName="main"
Jul 13 19:33:21.883320 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:33:21.883282    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="62328ddb-2423-4182-9a77-b0d7c284f208" containerName="main"
Jul 13 19:33:21.883631 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:33:21.883345    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="62328ddb-2423-4182-9a77-b0d7c284f208" containerName="llm-d-routing-sidecar"
Jul 13 19:33:21.883631 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:33:21.883355    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f5a7e70-d1ae-4938-bd11-e286e8bac45a" containerName="main"
Jul 13 19:33:21.883631 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:33:21.883362    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="62328ddb-2423-4182-9a77-b0d7c284f208" containerName="main"
Jul 13 19:33:21.886669 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:33:21.886652    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mhsdz"
Jul 13 19:33:21.889665 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:33:21.889640    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-pvc-router-manage-e8706282-kserve-dockercfg-68275\""
Jul 13 19:33:21.889799 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:33:21.889732    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvb19f98874e050eec8ca94d49676113f0-kserve-self-signed-certs\""
Jul 13 19:33:21.899097 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:33:21.899074    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mhsdz"]
Jul 13 19:33:21.901061 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:33:21.901036    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/762ad0f9-19a6-4a36-b4eb-a50f7291ab13-home\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mhsdz\" (UID: \"762ad0f9-19a6-4a36-b4eb-a50f7291ab13\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mhsdz"
Jul 13 19:33:21.901174 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:33:21.901075    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/762ad0f9-19a6-4a36-b4eb-a50f7291ab13-dshm\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mhsdz\" (UID: \"762ad0f9-19a6-4a36-b4eb-a50f7291ab13\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mhsdz"
Jul 13 19:33:21.901174 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:33:21.901165    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/762ad0f9-19a6-4a36-b4eb-a50f7291ab13-model-cache\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mhsdz\" (UID: \"762ad0f9-19a6-4a36-b4eb-a50f7291ab13\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mhsdz"
Jul 13 19:33:21.901280 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:33:21.901191    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/762ad0f9-19a6-4a36-b4eb-a50f7291ab13-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mhsdz\" (UID: \"762ad0f9-19a6-4a36-b4eb-a50f7291ab13\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mhsdz"
Jul 13 19:33:21.901280 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:33:21.901229    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6x8dq\" (UniqueName: \"kubernetes.io/projected/762ad0f9-19a6-4a36-b4eb-a50f7291ab13-kube-api-access-6x8dq\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mhsdz\" (UID: \"762ad0f9-19a6-4a36-b4eb-a50f7291ab13\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mhsdz"
Jul 13 19:33:21.901280 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:33:21.901257    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/762ad0f9-19a6-4a36-b4eb-a50f7291ab13-tls-certs\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mhsdz\" (UID: \"762ad0f9-19a6-4a36-b4eb-a50f7291ab13\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mhsdz"
Jul 13 19:33:22.002297 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:33:22.002262    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/762ad0f9-19a6-4a36-b4eb-a50f7291ab13-model-cache\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mhsdz\" (UID: \"762ad0f9-19a6-4a36-b4eb-a50f7291ab13\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mhsdz"
Jul 13 19:33:22.002480 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:33:22.002306    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/762ad0f9-19a6-4a36-b4eb-a50f7291ab13-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mhsdz\" (UID: \"762ad0f9-19a6-4a36-b4eb-a50f7291ab13\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mhsdz"
Jul 13 19:33:22.002480 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:33:22.002364    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6x8dq\" (UniqueName: \"kubernetes.io/projected/762ad0f9-19a6-4a36-b4eb-a50f7291ab13-kube-api-access-6x8dq\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mhsdz\" (UID: \"762ad0f9-19a6-4a36-b4eb-a50f7291ab13\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mhsdz"
Jul 13 19:33:22.002480 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:33:22.002393    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/762ad0f9-19a6-4a36-b4eb-a50f7291ab13-tls-certs\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mhsdz\" (UID: \"762ad0f9-19a6-4a36-b4eb-a50f7291ab13\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mhsdz"
Jul 13 19:33:22.002480 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:33:22.002446    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/762ad0f9-19a6-4a36-b4eb-a50f7291ab13-home\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mhsdz\" (UID: \"762ad0f9-19a6-4a36-b4eb-a50f7291ab13\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mhsdz"
Jul 13 19:33:22.002685 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:33:22.002502    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/762ad0f9-19a6-4a36-b4eb-a50f7291ab13-dshm\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mhsdz\" (UID: \"762ad0f9-19a6-4a36-b4eb-a50f7291ab13\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mhsdz"
Jul 13 19:33:22.002754 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:33:22.002729    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/762ad0f9-19a6-4a36-b4eb-a50f7291ab13-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mhsdz\" (UID: \"762ad0f9-19a6-4a36-b4eb-a50f7291ab13\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mhsdz"
Jul 13 19:33:22.002810 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:33:22.002795    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/762ad0f9-19a6-4a36-b4eb-a50f7291ab13-model-cache\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mhsdz\" (UID: \"762ad0f9-19a6-4a36-b4eb-a50f7291ab13\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mhsdz"
Jul 13 19:33:22.002870 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:33:22.002836    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/762ad0f9-19a6-4a36-b4eb-a50f7291ab13-home\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mhsdz\" (UID: \"762ad0f9-19a6-4a36-b4eb-a50f7291ab13\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mhsdz"
Jul 13 19:33:22.005049 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:33:22.005020    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/762ad0f9-19a6-4a36-b4eb-a50f7291ab13-dshm\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mhsdz\" (UID: \"762ad0f9-19a6-4a36-b4eb-a50f7291ab13\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mhsdz"
Jul 13 19:33:22.005261 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:33:22.005244    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/762ad0f9-19a6-4a36-b4eb-a50f7291ab13-tls-certs\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mhsdz\" (UID: \"762ad0f9-19a6-4a36-b4eb-a50f7291ab13\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mhsdz"
Jul 13 19:33:22.010849 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:33:22.010824    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6x8dq\" (UniqueName: \"kubernetes.io/projected/762ad0f9-19a6-4a36-b4eb-a50f7291ab13-kube-api-access-6x8dq\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mhsdz\" (UID: \"762ad0f9-19a6-4a36-b4eb-a50f7291ab13\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mhsdz"
Jul 13 19:33:22.167642 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:33:22.167609    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8gwt"]
Jul 13 19:33:22.172739 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:33:22.172599    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8gwt"
Jul 13 19:33:22.183385 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:33:22.183038    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8gwt"]
Jul 13 19:33:22.304938 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:33:22.304892    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/74e7b5d4-9d6e-4736-98c3-09d48ed6dec2-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8gwt\" (UID: \"74e7b5d4-9d6e-4736-98c3-09d48ed6dec2\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8gwt"
Jul 13 19:33:22.304938 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:33:22.304939    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/74e7b5d4-9d6e-4736-98c3-09d48ed6dec2-tls-certs\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8gwt\" (UID: \"74e7b5d4-9d6e-4736-98c3-09d48ed6dec2\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8gwt"
Jul 13 19:33:22.305162 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:33:22.304963    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/74e7b5d4-9d6e-4736-98c3-09d48ed6dec2-home\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8gwt\" (UID: \"74e7b5d4-9d6e-4736-98c3-09d48ed6dec2\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8gwt"
Jul 13 19:33:22.305162 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:33:22.305054    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njklf\" (UniqueName: \"kubernetes.io/projected/74e7b5d4-9d6e-4736-98c3-09d48ed6dec2-kube-api-access-njklf\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8gwt\" (UID: \"74e7b5d4-9d6e-4736-98c3-09d48ed6dec2\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8gwt"
Jul 13 19:33:22.305162 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:33:22.305114    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/74e7b5d4-9d6e-4736-98c3-09d48ed6dec2-model-cache\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8gwt\" (UID: \"74e7b5d4-9d6e-4736-98c3-09d48ed6dec2\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8gwt"
Jul 13 19:33:22.305309 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:33:22.305208    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/74e7b5d4-9d6e-4736-98c3-09d48ed6dec2-dshm\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8gwt\" (UID: \"74e7b5d4-9d6e-4736-98c3-09d48ed6dec2\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8gwt"
Jul 13 19:33:22.406299 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:33:22.406216    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/74e7b5d4-9d6e-4736-98c3-09d48ed6dec2-model-cache\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8gwt\" (UID: \"74e7b5d4-9d6e-4736-98c3-09d48ed6dec2\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8gwt"
Jul 13 19:33:22.406299 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:33:22.406295    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/74e7b5d4-9d6e-4736-98c3-09d48ed6dec2-dshm\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8gwt\" (UID: \"74e7b5d4-9d6e-4736-98c3-09d48ed6dec2\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8gwt"
Jul 13 19:33:22.406540 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:33:22.406345    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/74e7b5d4-9d6e-4736-98c3-09d48ed6dec2-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8gwt\" (UID: \"74e7b5d4-9d6e-4736-98c3-09d48ed6dec2\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8gwt"
Jul 13 19:33:22.406540 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:33:22.406362    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/74e7b5d4-9d6e-4736-98c3-09d48ed6dec2-tls-certs\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8gwt\" (UID: \"74e7b5d4-9d6e-4736-98c3-09d48ed6dec2\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8gwt"
Jul 13 19:33:22.406540 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:33:22.406378    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/74e7b5d4-9d6e-4736-98c3-09d48ed6dec2-home\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8gwt\" (UID: \"74e7b5d4-9d6e-4736-98c3-09d48ed6dec2\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8gwt"
Jul 13 19:33:22.406751 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:33:22.406529    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njklf\" (UniqueName: \"kubernetes.io/projected/74e7b5d4-9d6e-4736-98c3-09d48ed6dec2-kube-api-access-njklf\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8gwt\" (UID: \"74e7b5d4-9d6e-4736-98c3-09d48ed6dec2\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8gwt"
Jul 13 19:33:22.406811 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:33:22.406763    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/74e7b5d4-9d6e-4736-98c3-09d48ed6dec2-model-cache\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8gwt\" (UID: \"74e7b5d4-9d6e-4736-98c3-09d48ed6dec2\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8gwt"
Jul 13 19:33:22.406870 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:33:22.406816    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/74e7b5d4-9d6e-4736-98c3-09d48ed6dec2-home\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8gwt\" (UID: \"74e7b5d4-9d6e-4736-98c3-09d48ed6dec2\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8gwt"
Jul 13 19:33:22.406870 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:33:22.406838    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/74e7b5d4-9d6e-4736-98c3-09d48ed6dec2-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8gwt\" (UID: \"74e7b5d4-9d6e-4736-98c3-09d48ed6dec2\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8gwt"
Jul 13 19:33:22.408752 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:33:22.408730    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/74e7b5d4-9d6e-4736-98c3-09d48ed6dec2-dshm\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8gwt\" (UID: \"74e7b5d4-9d6e-4736-98c3-09d48ed6dec2\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8gwt"
Jul 13 19:33:22.408929 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:33:22.408908    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/74e7b5d4-9d6e-4736-98c3-09d48ed6dec2-tls-certs\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8gwt\" (UID: \"74e7b5d4-9d6e-4736-98c3-09d48ed6dec2\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8gwt"
Jul 13 19:33:22.424679 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:33:22.424655    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-njklf\" (UniqueName: \"kubernetes.io/projected/74e7b5d4-9d6e-4736-98c3-09d48ed6dec2-kube-api-access-njklf\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8gwt\" (UID: \"74e7b5d4-9d6e-4736-98c3-09d48ed6dec2\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8gwt"
Jul 13 19:33:50.327428 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:33:50.327341    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-664bd66db5-cjjg6"]
Jul 13 19:33:50.331332 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:33:50.331309    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-664bd66db5-cjjg6"
Jul 13 19:33:50.334290 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:33:50.334266    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 13 19:33:50.334422 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:33:50.334268    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-with-refs-pd-test-kserve-self-signed-certs\""
Jul 13 19:33:50.334422 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:33:50.334313    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-with-refs-pd-test-kserve-dockercfg-2ccj7\""
Jul 13 19:33:50.349971 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:33:50.349944    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-664bd66db5-cjjg6"]
Jul 13 19:33:50.358522 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:33:50.358498    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-9d68b57d4-trssd"]
Jul 13 19:33:50.362079 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:33:50.362062    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-9d68b57d4-trssd"
Jul 13 19:33:50.375003 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:33:50.374984    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-9d68b57d4-trssd"]
Jul 13 19:33:50.487683 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:33:50.487653    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3641c4ec-f761-489e-87f0-26a92715d84e-tmp-dir\") pod \"router-with-refs-pd-test-kserve-664bd66db5-cjjg6\" (UID: \"3641c4ec-f761-489e-87f0-26a92715d84e\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-664bd66db5-cjjg6"
Jul 13 19:33:50.487866 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:33:50.487697    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e36bb45-cdf3-44e8-8b67-8bab1385aa99-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-9d68b57d4-trssd\" (UID: \"5e36bb45-cdf3-44e8-8b67-8bab1385aa99\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-9d68b57d4-trssd"
Jul 13 19:33:50.487866 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:33:50.487770    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e36bb45-cdf3-44e8-8b67-8bab1385aa99-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-9d68b57d4-trssd\" (UID: \"5e36bb45-cdf3-44e8-8b67-8bab1385aa99\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-9d68b57d4-trssd"
Jul 13 19:33:50.487866 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:33:50.487795    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3641c4ec-f761-489e-87f0-26a92715d84e-model-cache\") pod \"router-with-refs-pd-test-kserve-664bd66db5-cjjg6\" (UID: \"3641c4ec-f761-489e-87f0-26a92715d84e\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-664bd66db5-cjjg6"
Jul 13 19:33:50.487866 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:33:50.487847    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3641c4ec-f761-489e-87f0-26a92715d84e-tls-certs\") pod \"router-with-refs-pd-test-kserve-664bd66db5-cjjg6\" (UID: \"3641c4ec-f761-489e-87f0-26a92715d84e\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-664bd66db5-cjjg6"
Jul 13 19:33:50.488010 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:33:50.487908    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3641c4ec-f761-489e-87f0-26a92715d84e-dshm\") pod \"router-with-refs-pd-test-kserve-664bd66db5-cjjg6\" (UID: \"3641c4ec-f761-489e-87f0-26a92715d84e\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-664bd66db5-cjjg6"
Jul 13 19:33:50.488010 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:33:50.487947    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5e36bb45-cdf3-44e8-8b67-8bab1385aa99-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-9d68b57d4-trssd\" (UID: \"5e36bb45-cdf3-44e8-8b67-8bab1385aa99\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-9d68b57d4-trssd"
Jul 13 19:33:50.488010 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:33:50.487966    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l445j\" (UniqueName: \"kubernetes.io/projected/3641c4ec-f761-489e-87f0-26a92715d84e-kube-api-access-l445j\") pod \"router-with-refs-pd-test-kserve-664bd66db5-cjjg6\" (UID: \"3641c4ec-f761-489e-87f0-26a92715d84e\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-664bd66db5-cjjg6"
Jul 13 19:33:50.488010 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:33:50.487996    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3641c4ec-f761-489e-87f0-26a92715d84e-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-664bd66db5-cjjg6\" (UID: \"3641c4ec-f761-489e-87f0-26a92715d84e\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-664bd66db5-cjjg6"
Jul 13 19:33:50.488144 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:33:50.488024    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8l62r\" (UniqueName: \"kubernetes.io/projected/5e36bb45-cdf3-44e8-8b67-8bab1385aa99-kube-api-access-8l62r\") pod \"router-with-refs-pd-test-kserve-prefill-9d68b57d4-trssd\" (UID: \"5e36bb45-cdf3-44e8-8b67-8bab1385aa99\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-9d68b57d4-trssd"
Jul 13 19:33:50.488144 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:33:50.488044    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e36bb45-cdf3-44e8-8b67-8bab1385aa99-tmp-dir\") pod \"router-with-refs-pd-test-kserve-prefill-9d68b57d4-trssd\" (UID: \"5e36bb45-cdf3-44e8-8b67-8bab1385aa99\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-9d68b57d4-trssd"
Jul 13 19:33:50.488144 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:33:50.488075    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e36bb45-cdf3-44e8-8b67-8bab1385aa99-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-9d68b57d4-trssd\" (UID: \"5e36bb45-cdf3-44e8-8b67-8bab1385aa99\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-9d68b57d4-trssd"
Jul 13 19:33:50.488144 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:33:50.488122    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e36bb45-cdf3-44e8-8b67-8bab1385aa99-home\") pod \"router-with-refs-pd-test-kserve-prefill-9d68b57d4-trssd\" (UID: \"5e36bb45-cdf3-44e8-8b67-8bab1385aa99\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-9d68b57d4-trssd"
Jul 13 19:33:50.488144 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:33:50.488142    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3641c4ec-f761-489e-87f0-26a92715d84e-home\") pod \"router-with-refs-pd-test-kserve-664bd66db5-cjjg6\" (UID: \"3641c4ec-f761-489e-87f0-26a92715d84e\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-664bd66db5-cjjg6"
Jul 13 19:33:50.589499 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:33:50.589418    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3641c4ec-f761-489e-87f0-26a92715d84e-tmp-dir\") pod \"router-with-refs-pd-test-kserve-664bd66db5-cjjg6\" (UID: \"3641c4ec-f761-489e-87f0-26a92715d84e\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-664bd66db5-cjjg6"
Jul 13 19:33:50.589499 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:33:50.589455    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e36bb45-cdf3-44e8-8b67-8bab1385aa99-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-9d68b57d4-trssd\" (UID: \"5e36bb45-cdf3-44e8-8b67-8bab1385aa99\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-9d68b57d4-trssd"
Jul 13 19:33:50.589499 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:33:50.589498    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e36bb45-cdf3-44e8-8b67-8bab1385aa99-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-9d68b57d4-trssd\" (UID: \"5e36bb45-cdf3-44e8-8b67-8bab1385aa99\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-9d68b57d4-trssd"
Jul 13 19:33:50.589802 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:33:50.589516    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3641c4ec-f761-489e-87f0-26a92715d84e-model-cache\") pod \"router-with-refs-pd-test-kserve-664bd66db5-cjjg6\" (UID: \"3641c4ec-f761-489e-87f0-26a92715d84e\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-664bd66db5-cjjg6"
Jul 13 19:33:50.589802 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:33:50.589566    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3641c4ec-f761-489e-87f0-26a92715d84e-tls-certs\") pod \"router-with-refs-pd-test-kserve-664bd66db5-cjjg6\" (UID: \"3641c4ec-f761-489e-87f0-26a92715d84e\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-664bd66db5-cjjg6"
Jul 13 19:33:50.589802 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:33:50.589589    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3641c4ec-f761-489e-87f0-26a92715d84e-dshm\") pod \"router-with-refs-pd-test-kserve-664bd66db5-cjjg6\" (UID: \"3641c4ec-f761-489e-87f0-26a92715d84e\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-664bd66db5-cjjg6"
Jul 13 19:33:50.589802 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:33:50.589615    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5e36bb45-cdf3-44e8-8b67-8bab1385aa99-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-9d68b57d4-trssd\" (UID: \"5e36bb45-cdf3-44e8-8b67-8bab1385aa99\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-9d68b57d4-trssd"
Jul 13 19:33:50.589802 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:33:50.589632    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l445j\" (UniqueName: \"kubernetes.io/projected/3641c4ec-f761-489e-87f0-26a92715d84e-kube-api-access-l445j\") pod \"router-with-refs-pd-test-kserve-664bd66db5-cjjg6\" (UID: \"3641c4ec-f761-489e-87f0-26a92715d84e\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-664bd66db5-cjjg6"
Jul 13 19:33:50.589802 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:33:50.589663    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3641c4ec-f761-489e-87f0-26a92715d84e-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-664bd66db5-cjjg6\" (UID: \"3641c4ec-f761-489e-87f0-26a92715d84e\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-664bd66db5-cjjg6"
Jul 13 19:33:50.589802 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:33:50.589695    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8l62r\" (UniqueName: \"kubernetes.io/projected/5e36bb45-cdf3-44e8-8b67-8bab1385aa99-kube-api-access-8l62r\") pod \"router-with-refs-pd-test-kserve-prefill-9d68b57d4-trssd\" (UID: \"5e36bb45-cdf3-44e8-8b67-8bab1385aa99\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-9d68b57d4-trssd"
Jul 13 19:33:50.589802 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:33:50.589744    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e36bb45-cdf3-44e8-8b67-8bab1385aa99-tmp-dir\") pod \"router-with-refs-pd-test-kserve-prefill-9d68b57d4-trssd\" (UID: \"5e36bb45-cdf3-44e8-8b67-8bab1385aa99\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-9d68b57d4-trssd"
Jul 13 19:33:50.589802 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:33:50.589781    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e36bb45-cdf3-44e8-8b67-8bab1385aa99-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-9d68b57d4-trssd\" (UID: \"5e36bb45-cdf3-44e8-8b67-8bab1385aa99\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-9d68b57d4-trssd"
Jul 13 19:33:50.590252 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:33:50.589854    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e36bb45-cdf3-44e8-8b67-8bab1385aa99-home\") pod \"router-with-refs-pd-test-kserve-prefill-9d68b57d4-trssd\" (UID: \"5e36bb45-cdf3-44e8-8b67-8bab1385aa99\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-9d68b57d4-trssd"
Jul 13 19:33:50.590252 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:33:50.589885    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3641c4ec-f761-489e-87f0-26a92715d84e-home\") pod \"router-with-refs-pd-test-kserve-664bd66db5-cjjg6\" (UID: \"3641c4ec-f761-489e-87f0-26a92715d84e\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-664bd66db5-cjjg6"
Jul 13 19:33:50.590252 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:33:50.589934    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e36bb45-cdf3-44e8-8b67-8bab1385aa99-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-9d68b57d4-trssd\" (UID: \"5e36bb45-cdf3-44e8-8b67-8bab1385aa99\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-9d68b57d4-trssd"
Jul 13 19:33:50.590252 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:33:50.589985    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5e36bb45-cdf3-44e8-8b67-8bab1385aa99-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-9d68b57d4-trssd\" (UID: \"5e36bb45-cdf3-44e8-8b67-8bab1385aa99\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-9d68b57d4-trssd"
Jul 13 19:33:50.590252 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:33:50.590019    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3641c4ec-f761-489e-87f0-26a92715d84e-model-cache\") pod \"router-with-refs-pd-test-kserve-664bd66db5-cjjg6\" (UID: \"3641c4ec-f761-489e-87f0-26a92715d84e\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-664bd66db5-cjjg6"
Jul 13 19:33:50.590252 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:33:50.590199    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3641c4ec-f761-489e-87f0-26a92715d84e-home\") pod \"router-with-refs-pd-test-kserve-664bd66db5-cjjg6\" (UID: \"3641c4ec-f761-489e-87f0-26a92715d84e\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-664bd66db5-cjjg6"
Jul 13 19:33:50.590252 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:33:50.590201    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e36bb45-cdf3-44e8-8b67-8bab1385aa99-tmp-dir\") pod \"router-with-refs-pd-test-kserve-prefill-9d68b57d4-trssd\" (UID: \"5e36bb45-cdf3-44e8-8b67-8bab1385aa99\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-9d68b57d4-trssd"
Jul 13 19:33:50.590816 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:33:50.590283    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3641c4ec-f761-489e-87f0-26a92715d84e-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-664bd66db5-cjjg6\" (UID: \"3641c4ec-f761-489e-87f0-26a92715d84e\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-664bd66db5-cjjg6"
Jul 13 19:33:50.590816 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:33:50.590334    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3641c4ec-f761-489e-87f0-26a92715d84e-tmp-dir\") pod \"router-with-refs-pd-test-kserve-664bd66db5-cjjg6\" (UID: \"3641c4ec-f761-489e-87f0-26a92715d84e\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-664bd66db5-cjjg6"
Jul 13 19:33:50.590816 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:33:50.590422    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e36bb45-cdf3-44e8-8b67-8bab1385aa99-home\") pod \"router-with-refs-pd-test-kserve-prefill-9d68b57d4-trssd\" (UID: \"5e36bb45-cdf3-44e8-8b67-8bab1385aa99\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-9d68b57d4-trssd"
Jul 13 19:33:50.592190 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:33:50.592160    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3641c4ec-f761-489e-87f0-26a92715d84e-dshm\") pod \"router-with-refs-pd-test-kserve-664bd66db5-cjjg6\" (UID: \"3641c4ec-f761-489e-87f0-26a92715d84e\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-664bd66db5-cjjg6"
Jul 13 19:33:50.592306 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:33:50.592254    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e36bb45-cdf3-44e8-8b67-8bab1385aa99-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-9d68b57d4-trssd\" (UID: \"5e36bb45-cdf3-44e8-8b67-8bab1385aa99\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-9d68b57d4-trssd"
Jul 13 19:33:50.592506 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:33:50.592489    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3641c4ec-f761-489e-87f0-26a92715d84e-tls-certs\") pod \"router-with-refs-pd-test-kserve-664bd66db5-cjjg6\" (UID: \"3641c4ec-f761-489e-87f0-26a92715d84e\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-664bd66db5-cjjg6"
Jul 13 19:33:50.592856 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:33:50.592837    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e36bb45-cdf3-44e8-8b67-8bab1385aa99-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-9d68b57d4-trssd\" (UID: \"5e36bb45-cdf3-44e8-8b67-8bab1385aa99\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-9d68b57d4-trssd"
Jul 13 19:33:50.598261 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:33:50.598235    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8l62r\" (UniqueName: \"kubernetes.io/projected/5e36bb45-cdf3-44e8-8b67-8bab1385aa99-kube-api-access-8l62r\") pod \"router-with-refs-pd-test-kserve-prefill-9d68b57d4-trssd\" (UID: \"5e36bb45-cdf3-44e8-8b67-8bab1385aa99\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-9d68b57d4-trssd"
Jul 13 19:33:50.598365 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:33:50.598239    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l445j\" (UniqueName: \"kubernetes.io/projected/3641c4ec-f761-489e-87f0-26a92715d84e-kube-api-access-l445j\") pod \"router-with-refs-pd-test-kserve-664bd66db5-cjjg6\" (UID: \"3641c4ec-f761-489e-87f0-26a92715d84e\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-664bd66db5-cjjg6"
Jul 13 19:33:50.642305 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:33:50.642281    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-664bd66db5-cjjg6"
Jul 13 19:33:50.672627 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:33:50.672593    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-9d68b57d4-trssd"
Jul 13 19:33:50.796545 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:33:50.796515    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-664bd66db5-cjjg6"]
Jul 13 19:33:50.798041 ip-10-0-128-215 kubenswrapper[2565]: W0713 19:33:50.798012    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3641c4ec_f761_489e_87f0_26a92715d84e.slice/crio-db25a18cf277d97aecfb8e1c886b8ff84d7eeb8edfe81eb68b8f046ac4a3f414 WatchSource:0}: Error finding container db25a18cf277d97aecfb8e1c886b8ff84d7eeb8edfe81eb68b8f046ac4a3f414: Status 404 returned error can't find the container with id db25a18cf277d97aecfb8e1c886b8ff84d7eeb8edfe81eb68b8f046ac4a3f414
Jul 13 19:33:50.825359 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:33:50.825336    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-9d68b57d4-trssd"]
Jul 13 19:33:50.829967 ip-10-0-128-215 kubenswrapper[2565]: W0713 19:33:50.829934    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e36bb45_cdf3_44e8_8b67_8bab1385aa99.slice/crio-3789a29cd31f31dcff9fc7350dc13f95a39e58c1b731c000217806e27a6a214b WatchSource:0}: Error finding container 3789a29cd31f31dcff9fc7350dc13f95a39e58c1b731c000217806e27a6a214b: Status 404 returned error can't find the container with id 3789a29cd31f31dcff9fc7350dc13f95a39e58c1b731c000217806e27a6a214b
Jul 13 19:33:51.765194 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:33:51.765067    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-664bd66db5-cjjg6" event={"ID":"3641c4ec-f761-489e-87f0-26a92715d84e","Type":"ContainerStarted","Data":"b70ebd173b629a5bf6977783af3790bf844d9769aa4aa1286919e4a9ed93bcc6"}
Jul 13 19:33:51.765194 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:33:51.765142    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-664bd66db5-cjjg6" event={"ID":"3641c4ec-f761-489e-87f0-26a92715d84e","Type":"ContainerStarted","Data":"db25a18cf277d97aecfb8e1c886b8ff84d7eeb8edfe81eb68b8f046ac4a3f414"}
Jul 13 19:33:51.765932 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:33:51.765292    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-664bd66db5-cjjg6"
Jul 13 19:33:51.767119 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:33:51.767095    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-9d68b57d4-trssd" event={"ID":"5e36bb45-cdf3-44e8-8b67-8bab1385aa99","Type":"ContainerStarted","Data":"3d995604e7069e8a7ca5bc3441c5b188b8f7ec45ac9e1639c2c7455dbdda7a6b"}
Jul 13 19:33:51.767250 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:33:51.767184    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-9d68b57d4-trssd" event={"ID":"5e36bb45-cdf3-44e8-8b67-8bab1385aa99","Type":"ContainerStarted","Data":"3789a29cd31f31dcff9fc7350dc13f95a39e58c1b731c000217806e27a6a214b"}
Jul 13 19:33:52.774735 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:33:52.774668    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-664bd66db5-cjjg6" event={"ID":"3641c4ec-f761-489e-87f0-26a92715d84e","Type":"ContainerStarted","Data":"b97dd6145053085cdf2a73c44e217b8ef3cbe7acc36faaa702e95f6beedabf9e"}
Jul 13 19:33:54.785142 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:33:54.785106    2565 generic.go:356] "Generic (PLEG): container finished" podID="5e36bb45-cdf3-44e8-8b67-8bab1385aa99" containerID="3d995604e7069e8a7ca5bc3441c5b188b8f7ec45ac9e1639c2c7455dbdda7a6b" exitCode=0
Jul 13 19:33:54.785553 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:33:54.785180    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-9d68b57d4-trssd" event={"ID":"5e36bb45-cdf3-44e8-8b67-8bab1385aa99","Type":"ContainerDied","Data":"3d995604e7069e8a7ca5bc3441c5b188b8f7ec45ac9e1639c2c7455dbdda7a6b"}
Jul 13 19:33:55.793168 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:33:55.793127    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-9d68b57d4-trssd" event={"ID":"5e36bb45-cdf3-44e8-8b67-8bab1385aa99","Type":"ContainerStarted","Data":"e2a78c498e6f9c98b0c890fd7da7400ca1720a1bdc18bb3c16792ab9a5595e47"}
Jul 13 19:33:55.795236 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:33:55.795204    2565 generic.go:356] "Generic (PLEG): container finished" podID="3641c4ec-f761-489e-87f0-26a92715d84e" containerID="b97dd6145053085cdf2a73c44e217b8ef3cbe7acc36faaa702e95f6beedabf9e" exitCode=0
Jul 13 19:33:55.795376 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:33:55.795278    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-664bd66db5-cjjg6" event={"ID":"3641c4ec-f761-489e-87f0-26a92715d84e","Type":"ContainerDied","Data":"b97dd6145053085cdf2a73c44e217b8ef3cbe7acc36faaa702e95f6beedabf9e"}
Jul 13 19:33:55.819001 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:33:55.818905    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-9d68b57d4-trssd" podStartSLOduration=1.989343298 podStartE2EDuration="5.818887229s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.829543931s" isStatefulPod=false podCreationTimestamp="2026-07-13 19:33:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 19:33:55.816044986 +0000 UTC m=+2622.341178423" watchObservedRunningTime="2026-07-13 19:33:55.818887229 +0000 UTC m=+2622.344020647"
Jul 13 19:33:56.803805 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:33:56.803770    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-664bd66db5-cjjg6" event={"ID":"3641c4ec-f761-489e-87f0-26a92715d84e","Type":"ContainerStarted","Data":"f1d49422b184040fe76d8441320324d91d54121081c4414cf1ff80cbf4011daa"}
Jul 13 19:33:56.829549 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:33:56.829494    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-664bd66db5-cjjg6" podStartSLOduration=3.130154194 podStartE2EDuration="6.829477733s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.699323539s" isStatefulPod=false podCreationTimestamp="2026-07-13 19:33:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 19:33:56.826558451 +0000 UTC m=+2623.351691867" watchObservedRunningTime="2026-07-13 19:33:56.829477733 +0000 UTC m=+2623.354611148"
Jul 13 19:34:00.642532 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:34:00.642494    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-664bd66db5-cjjg6"
Jul 13 19:34:00.643064 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:34:00.642571    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-664bd66db5-cjjg6"
Jul 13 19:34:00.643942 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:34:00.643904    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-664bd66db5-cjjg6" podUID="3641c4ec-f761-489e-87f0-26a92715d84e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8001/health\": dial tcp 10.134.0.59:8001: connect: connection refused"
Jul 13 19:34:00.662154 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:34:00.662129    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-664bd66db5-cjjg6"
Jul 13 19:34:00.673335 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:34:00.673306    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-9d68b57d4-trssd"
Jul 13 19:34:00.673639 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:34:00.673616    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-9d68b57d4-trssd"
Jul 13 19:34:00.674773 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:34:00.674745    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-9d68b57d4-trssd" podUID="5e36bb45-cdf3-44e8-8b67-8bab1385aa99" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": dial tcp 10.134.0.60:8000: connect: connection refused"
Jul 13 19:34:10.643256 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:34:10.643205    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-664bd66db5-cjjg6" podUID="3641c4ec-f761-489e-87f0-26a92715d84e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8001/health\": dial tcp 10.134.0.59:8001: connect: connection refused"
Jul 13 19:34:10.673156 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:34:10.673113    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-9d68b57d4-trssd" podUID="5e36bb45-cdf3-44e8-8b67-8bab1385aa99" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": dial tcp 10.134.0.60:8000: connect: connection refused"
Jul 13 19:34:20.283086 ip-10-0-128-215 kubenswrapper[2565]: E0713 19:34:20.283030    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dmww7f" podUID="634b78c4-751c-4880-9f6c-1ea69b737987"
Jul 13 19:34:20.643447 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:34:20.643323    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-664bd66db5-cjjg6" podUID="3641c4ec-f761-489e-87f0-26a92715d84e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8001/health\": dial tcp 10.134.0.59:8001: connect: connection refused"
Jul 13 19:34:20.673076 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:34:20.673031    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-9d68b57d4-trssd" podUID="5e36bb45-cdf3-44e8-8b67-8bab1385aa99" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": dial tcp 10.134.0.60:8000: connect: connection refused"
Jul 13 19:34:20.946739 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:34:20.946637    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dmww7f"
Jul 13 19:34:30.643033 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:34:30.642983    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-664bd66db5-cjjg6" podUID="3641c4ec-f761-489e-87f0-26a92715d84e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8001/health\": dial tcp 10.134.0.59:8001: connect: connection refused"
Jul 13 19:34:30.672978 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:34:30.672937    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-9d68b57d4-trssd" podUID="5e36bb45-cdf3-44e8-8b67-8bab1385aa99" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": dial tcp 10.134.0.60:8000: connect: connection refused"
Jul 13 19:34:40.642936 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:34:40.642887    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-664bd66db5-cjjg6" podUID="3641c4ec-f761-489e-87f0-26a92715d84e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8001/health\": dial tcp 10.134.0.59:8001: connect: connection refused"
Jul 13 19:34:40.673827 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:34:40.673788    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-9d68b57d4-trssd" podUID="5e36bb45-cdf3-44e8-8b67-8bab1385aa99" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": dial tcp 10.134.0.60:8000: connect: connection refused"
Jul 13 19:34:50.643021 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:34:50.642969    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-664bd66db5-cjjg6" podUID="3641c4ec-f761-489e-87f0-26a92715d84e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8001/health\": dial tcp 10.134.0.59:8001: connect: connection refused"
Jul 13 19:34:50.673070 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:34:50.673025    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-9d68b57d4-trssd" podUID="5e36bb45-cdf3-44e8-8b67-8bab1385aa99" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": dial tcp 10.134.0.60:8000: connect: connection refused"
Jul 13 19:35:00.643605 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:35:00.643557    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-664bd66db5-cjjg6" podUID="3641c4ec-f761-489e-87f0-26a92715d84e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8001/health\": dial tcp 10.134.0.59:8001: connect: connection refused"
Jul 13 19:35:00.673801 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:35:00.673761    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-9d68b57d4-trssd" podUID="5e36bb45-cdf3-44e8-8b67-8bab1385aa99" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": dial tcp 10.134.0.60:8000: connect: connection refused"
Jul 13 19:35:10.643447 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:35:10.643329    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-664bd66db5-cjjg6" podUID="3641c4ec-f761-489e-87f0-26a92715d84e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8001/health\": dial tcp 10.134.0.59:8001: connect: connection refused"
Jul 13 19:35:10.673774 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:35:10.673735    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-9d68b57d4-trssd" podUID="5e36bb45-cdf3-44e8-8b67-8bab1385aa99" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": dial tcp 10.134.0.60:8000: connect: connection refused"
Jul 13 19:35:14.149041 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:35:14.149002    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2bscc_2f0655bd-4a09-4314-8c1b-e30191eaec0e/ovn-acl-logging/0.log"
Jul 13 19:35:14.151836 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:35:14.151812    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2bscc_2f0655bd-4a09-4314-8c1b-e30191eaec0e/ovn-acl-logging/0.log"
Jul 13 19:35:20.642839 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:35:20.642798    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-664bd66db5-cjjg6" podUID="3641c4ec-f761-489e-87f0-26a92715d84e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8001/health\": dial tcp 10.134.0.59:8001: connect: connection refused"
Jul 13 19:35:20.673076 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:35:20.673034    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-9d68b57d4-trssd" podUID="5e36bb45-cdf3-44e8-8b67-8bab1385aa99" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": dial tcp 10.134.0.60:8000: connect: connection refused"
Jul 13 19:35:24.896893 ip-10-0-128-215 kubenswrapper[2565]: E0713 19:35:24.896840    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mhsdz" podUID="762ad0f9-19a6-4a36-b4eb-a50f7291ab13"
Jul 13 19:35:25.187553 ip-10-0-128-215 kubenswrapper[2565]: E0713 19:35:25.187445    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8gwt" podUID="74e7b5d4-9d6e-4736-98c3-09d48ed6dec2"
Jul 13 19:35:25.329052 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:35:25.329016    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8gwt"
Jul 13 19:35:25.329272 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:35:25.329017    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mhsdz"
Jul 13 19:35:30.643001 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:35:30.642948    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-664bd66db5-cjjg6" podUID="3641c4ec-f761-489e-87f0-26a92715d84e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8001/health\": dial tcp 10.134.0.59:8001: connect: connection refused"
Jul 13 19:35:30.673159 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:35:30.673122    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-9d68b57d4-trssd" podUID="5e36bb45-cdf3-44e8-8b67-8bab1385aa99" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": dial tcp 10.134.0.60:8000: connect: connection refused"
Jul 13 19:35:40.643715 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:35:40.643644    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-664bd66db5-cjjg6" podUID="3641c4ec-f761-489e-87f0-26a92715d84e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8001/health\": dial tcp 10.134.0.59:8001: connect: connection refused"
Jul 13 19:35:40.673248 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:35:40.673207    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-9d68b57d4-trssd" podUID="5e36bb45-cdf3-44e8-8b67-8bab1385aa99" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": dial tcp 10.134.0.60:8000: connect: connection refused"
Jul 13 19:35:50.643137 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:35:50.643080    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-664bd66db5-cjjg6" podUID="3641c4ec-f761-489e-87f0-26a92715d84e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8001/health\": dial tcp 10.134.0.59:8001: connect: connection refused"
Jul 13 19:35:50.673876 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:35:50.673824    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-9d68b57d4-trssd" podUID="5e36bb45-cdf3-44e8-8b67-8bab1385aa99" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": dial tcp 10.134.0.60:8000: connect: connection refused"
Jul 13 19:36:00.643073 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:36:00.643024    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-664bd66db5-cjjg6" podUID="3641c4ec-f761-489e-87f0-26a92715d84e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8001/health\": dial tcp 10.134.0.59:8001: connect: connection refused"
Jul 13 19:36:00.673312 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:36:00.673270    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-9d68b57d4-trssd" podUID="5e36bb45-cdf3-44e8-8b67-8bab1385aa99" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": dial tcp 10.134.0.60:8000: connect: connection refused"
Jul 13 19:36:10.642875 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:36:10.642809    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-664bd66db5-cjjg6" podUID="3641c4ec-f761-489e-87f0-26a92715d84e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8001/health\": dial tcp 10.134.0.59:8001: connect: connection refused"
Jul 13 19:36:10.673176 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:36:10.673134    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-9d68b57d4-trssd" podUID="5e36bb45-cdf3-44e8-8b67-8bab1385aa99" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": dial tcp 10.134.0.60:8000: connect: connection refused"
Jul 13 19:36:20.643593 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:36:20.643533    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-664bd66db5-cjjg6" podUID="3641c4ec-f761-489e-87f0-26a92715d84e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8001/health\": dial tcp 10.134.0.59:8001: connect: connection refused"
Jul 13 19:36:20.674087 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:36:20.674028    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-9d68b57d4-trssd" podUID="5e36bb45-cdf3-44e8-8b67-8bab1385aa99" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": dial tcp 10.134.0.60:8000: connect: connection refused"
Jul 13 19:36:23.948235 ip-10-0-128-215 kubenswrapper[2565]: E0713 19:36:23.948174    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dmww7f" podUID="634b78c4-751c-4880-9f6c-1ea69b737987"
Jul 13 19:36:24.654138 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:36:24.654099    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dmww7f"
Jul 13 19:36:30.643458 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:36:30.643401    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-664bd66db5-cjjg6" podUID="3641c4ec-f761-489e-87f0-26a92715d84e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8001/health\": dial tcp 10.134.0.59:8001: connect: connection refused"
Jul 13 19:36:30.673214 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:36:30.673171    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-9d68b57d4-trssd" podUID="5e36bb45-cdf3-44e8-8b67-8bab1385aa99" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": dial tcp 10.134.0.60:8000: connect: connection refused"
Jul 13 19:36:40.652918 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:36:40.652842    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-664bd66db5-cjjg6"
Jul 13 19:36:40.683347 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:36:40.683313    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-9d68b57d4-trssd"
Jul 13 19:36:40.692022 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:36:40.691999    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-9d68b57d4-trssd"
Jul 13 19:36:40.748235 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:36:40.748205    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-664bd66db5-cjjg6"
Jul 13 19:37:12.105831 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:37:12.105800    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-664bd66db5-cjjg6"]
Jul 13 19:37:12.108976 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:37:12.108947    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-9d68b57d4-trssd"]
Jul 13 19:37:27.134223 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:37:27.134159    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-9d68b57d4-trssd" podUID="5e36bb45-cdf3-44e8-8b67-8bab1385aa99" containerName="main" containerID="cri-o://e2a78c498e6f9c98b0c890fd7da7400ca1720a1bdc18bb3c16792ab9a5595e47" gracePeriod=45
Jul 13 19:37:27.138175 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:37:27.138124    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-664bd66db5-cjjg6" podUID="3641c4ec-f761-489e-87f0-26a92715d84e" containerName="main" containerID="cri-o://f1d49422b184040fe76d8441320324d91d54121081c4414cf1ff80cbf4011daa" gracePeriod=45
Jul 13 19:37:27.955968 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:37:27.955894    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-664bd66db5-cjjg6" podUID="3641c4ec-f761-489e-87f0-26a92715d84e" containerName="llm-d-routing-sidecar" containerID="cri-o://b70ebd173b629a5bf6977783af3790bf844d9769aa4aa1286919e4a9ed93bcc6" gracePeriod=45
Jul 13 19:37:27.975748 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:37:27.975689    2565 generic.go:356] "Generic (PLEG): container finished" podID="5e36bb45-cdf3-44e8-8b67-8bab1385aa99" containerID="e2a78c498e6f9c98b0c890fd7da7400ca1720a1bdc18bb3c16792ab9a5595e47" exitCode=0
Jul 13 19:37:27.975883 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:37:27.975740    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-9d68b57d4-trssd" event={"ID":"5e36bb45-cdf3-44e8-8b67-8bab1385aa99","Type":"ContainerDied","Data":"e2a78c498e6f9c98b0c890fd7da7400ca1720a1bdc18bb3c16792ab9a5595e47"}
Jul 13 19:37:27.978355 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:37:27.978330    2565 generic.go:356] "Generic (PLEG): container finished" podID="3641c4ec-f761-489e-87f0-26a92715d84e" containerID="f1d49422b184040fe76d8441320324d91d54121081c4414cf1ff80cbf4011daa" exitCode=0
Jul 13 19:37:27.978507 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:37:27.978377    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-664bd66db5-cjjg6" event={"ID":"3641c4ec-f761-489e-87f0-26a92715d84e","Type":"ContainerDied","Data":"f1d49422b184040fe76d8441320324d91d54121081c4414cf1ff80cbf4011daa"}
Jul 13 19:37:28.096624 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:37:28.096552    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-9d68b57d4-trssd"
Jul 13 19:37:28.204328 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:37:28.204304    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-664bd66db5-cjjg6"
Jul 13 19:37:28.263424 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:37:28.263396    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5e36bb45-cdf3-44e8-8b67-8bab1385aa99-kserve-provision-location\") pod \"5e36bb45-cdf3-44e8-8b67-8bab1385aa99\" (UID: \"5e36bb45-cdf3-44e8-8b67-8bab1385aa99\") "
Jul 13 19:37:28.263563 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:37:28.263435    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8l62r\" (UniqueName: \"kubernetes.io/projected/5e36bb45-cdf3-44e8-8b67-8bab1385aa99-kube-api-access-8l62r\") pod \"5e36bb45-cdf3-44e8-8b67-8bab1385aa99\" (UID: \"5e36bb45-cdf3-44e8-8b67-8bab1385aa99\") "
Jul 13 19:37:28.263563 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:37:28.263503    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e36bb45-cdf3-44e8-8b67-8bab1385aa99-tmp-dir\") pod \"5e36bb45-cdf3-44e8-8b67-8bab1385aa99\" (UID: \"5e36bb45-cdf3-44e8-8b67-8bab1385aa99\") "
Jul 13 19:37:28.263563 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:37:28.263534    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e36bb45-cdf3-44e8-8b67-8bab1385aa99-tls-certs\") pod \"5e36bb45-cdf3-44e8-8b67-8bab1385aa99\" (UID: \"5e36bb45-cdf3-44e8-8b67-8bab1385aa99\") "
Jul 13 19:37:28.263563 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:37:28.263557    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e36bb45-cdf3-44e8-8b67-8bab1385aa99-model-cache\") pod \"5e36bb45-cdf3-44e8-8b67-8bab1385aa99\" (UID: \"5e36bb45-cdf3-44e8-8b67-8bab1385aa99\") "
Jul 13 19:37:28.263834 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:37:28.263575    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e36bb45-cdf3-44e8-8b67-8bab1385aa99-home\") pod \"5e36bb45-cdf3-44e8-8b67-8bab1385aa99\" (UID: \"5e36bb45-cdf3-44e8-8b67-8bab1385aa99\") "
Jul 13 19:37:28.263834 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:37:28.263594    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e36bb45-cdf3-44e8-8b67-8bab1385aa99-dshm\") pod \"5e36bb45-cdf3-44e8-8b67-8bab1385aa99\" (UID: \"5e36bb45-cdf3-44e8-8b67-8bab1385aa99\") "
Jul 13 19:37:28.263940 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:37:28.263897    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e36bb45-cdf3-44e8-8b67-8bab1385aa99-model-cache" (OuterVolumeSpecName: "model-cache") pod "5e36bb45-cdf3-44e8-8b67-8bab1385aa99" (UID: "5e36bb45-cdf3-44e8-8b67-8bab1385aa99"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:37:28.264584 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:37:28.264550    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e36bb45-cdf3-44e8-8b67-8bab1385aa99-home" (OuterVolumeSpecName: "home") pod "5e36bb45-cdf3-44e8-8b67-8bab1385aa99" (UID: "5e36bb45-cdf3-44e8-8b67-8bab1385aa99"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:37:28.265886 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:37:28.265853    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e36bb45-cdf3-44e8-8b67-8bab1385aa99-kube-api-access-8l62r" (OuterVolumeSpecName: "kube-api-access-8l62r") pod "5e36bb45-cdf3-44e8-8b67-8bab1385aa99" (UID: "5e36bb45-cdf3-44e8-8b67-8bab1385aa99"). InnerVolumeSpecName "kube-api-access-8l62r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 19:37:28.265886 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:37:28.265859    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e36bb45-cdf3-44e8-8b67-8bab1385aa99-dshm" (OuterVolumeSpecName: "dshm") pod "5e36bb45-cdf3-44e8-8b67-8bab1385aa99" (UID: "5e36bb45-cdf3-44e8-8b67-8bab1385aa99"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:37:28.266014 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:37:28.265926    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e36bb45-cdf3-44e8-8b67-8bab1385aa99-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5e36bb45-cdf3-44e8-8b67-8bab1385aa99" (UID: "5e36bb45-cdf3-44e8-8b67-8bab1385aa99"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 19:37:28.276667 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:37:28.276635    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e36bb45-cdf3-44e8-8b67-8bab1385aa99-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5e36bb45-cdf3-44e8-8b67-8bab1385aa99" (UID: "5e36bb45-cdf3-44e8-8b67-8bab1385aa99"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:37:28.329556 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:37:28.329526    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e36bb45-cdf3-44e8-8b67-8bab1385aa99-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5e36bb45-cdf3-44e8-8b67-8bab1385aa99" (UID: "5e36bb45-cdf3-44e8-8b67-8bab1385aa99"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:37:28.329953 ip-10-0-128-215 kubenswrapper[2565]: E0713 19:37:28.329930    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8gwt" podUID="74e7b5d4-9d6e-4736-98c3-09d48ed6dec2"
Jul 13 19:37:28.330015 ip-10-0-128-215 kubenswrapper[2565]: E0713 19:37:28.329947    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mhsdz" podUID="762ad0f9-19a6-4a36-b4eb-a50f7291ab13"
Jul 13 19:37:28.364309 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:37:28.364250    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3641c4ec-f761-489e-87f0-26a92715d84e-tls-certs\") pod \"3641c4ec-f761-489e-87f0-26a92715d84e\" (UID: \"3641c4ec-f761-489e-87f0-26a92715d84e\") "
Jul 13 19:37:28.364309 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:37:28.364301    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3641c4ec-f761-489e-87f0-26a92715d84e-kserve-provision-location\") pod \"3641c4ec-f761-489e-87f0-26a92715d84e\" (UID: \"3641c4ec-f761-489e-87f0-26a92715d84e\") "
Jul 13 19:37:28.364442 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:37:28.364321    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3641c4ec-f761-489e-87f0-26a92715d84e-home\") pod \"3641c4ec-f761-489e-87f0-26a92715d84e\" (UID: \"3641c4ec-f761-489e-87f0-26a92715d84e\") "
Jul 13 19:37:28.364442 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:37:28.364348    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3641c4ec-f761-489e-87f0-26a92715d84e-dshm\") pod \"3641c4ec-f761-489e-87f0-26a92715d84e\" (UID: \"3641c4ec-f761-489e-87f0-26a92715d84e\") "
Jul 13 19:37:28.364442 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:37:28.364371    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3641c4ec-f761-489e-87f0-26a92715d84e-model-cache\") pod \"3641c4ec-f761-489e-87f0-26a92715d84e\" (UID: \"3641c4ec-f761-489e-87f0-26a92715d84e\") "
Jul 13 19:37:28.364442 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:37:28.364423    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l445j\" (UniqueName: \"kubernetes.io/projected/3641c4ec-f761-489e-87f0-26a92715d84e-kube-api-access-l445j\") pod \"3641c4ec-f761-489e-87f0-26a92715d84e\" (UID: \"3641c4ec-f761-489e-87f0-26a92715d84e\") "
Jul 13 19:37:28.364651 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:37:28.364453    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3641c4ec-f761-489e-87f0-26a92715d84e-tmp-dir\") pod \"3641c4ec-f761-489e-87f0-26a92715d84e\" (UID: \"3641c4ec-f761-489e-87f0-26a92715d84e\") "
Jul 13 19:37:28.364740 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:37:28.364659    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3641c4ec-f761-489e-87f0-26a92715d84e-model-cache" (OuterVolumeSpecName: "model-cache") pod "3641c4ec-f761-489e-87f0-26a92715d84e" (UID: "3641c4ec-f761-489e-87f0-26a92715d84e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:37:28.364953 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:37:28.364933    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3641c4ec-f761-489e-87f0-26a92715d84e-model-cache\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 13 19:37:28.365041 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:37:28.364961    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e36bb45-cdf3-44e8-8b67-8bab1385aa99-tmp-dir\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 13 19:37:28.365041 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:37:28.364974    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e36bb45-cdf3-44e8-8b67-8bab1385aa99-tls-certs\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 13 19:37:28.365041 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:37:28.364986    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e36bb45-cdf3-44e8-8b67-8bab1385aa99-model-cache\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 13 19:37:28.365041 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:37:28.364997    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e36bb45-cdf3-44e8-8b67-8bab1385aa99-home\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 13 19:37:28.365041 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:37:28.365009    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e36bb45-cdf3-44e8-8b67-8bab1385aa99-dshm\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 13 19:37:28.365041 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:37:28.365022    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5e36bb45-cdf3-44e8-8b67-8bab1385aa99-kserve-provision-location\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 13 19:37:28.365041 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:37:28.365035    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8l62r\" (UniqueName: \"kubernetes.io/projected/5e36bb45-cdf3-44e8-8b67-8bab1385aa99-kube-api-access-8l62r\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 13 19:37:28.365568 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:37:28.365143    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3641c4ec-f761-489e-87f0-26a92715d84e-home" (OuterVolumeSpecName: "home") pod "3641c4ec-f761-489e-87f0-26a92715d84e" (UID: "3641c4ec-f761-489e-87f0-26a92715d84e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:37:28.366661 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:37:28.366630    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3641c4ec-f761-489e-87f0-26a92715d84e-kube-api-access-l445j" (OuterVolumeSpecName: "kube-api-access-l445j") pod "3641c4ec-f761-489e-87f0-26a92715d84e" (UID: "3641c4ec-f761-489e-87f0-26a92715d84e"). InnerVolumeSpecName "kube-api-access-l445j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 19:37:28.366773 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:37:28.366697    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3641c4ec-f761-489e-87f0-26a92715d84e-dshm" (OuterVolumeSpecName: "dshm") pod "3641c4ec-f761-489e-87f0-26a92715d84e" (UID: "3641c4ec-f761-489e-87f0-26a92715d84e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:37:28.366875 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:37:28.366858    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3641c4ec-f761-489e-87f0-26a92715d84e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3641c4ec-f761-489e-87f0-26a92715d84e" (UID: "3641c4ec-f761-489e-87f0-26a92715d84e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 19:37:28.377826 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:37:28.377805    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3641c4ec-f761-489e-87f0-26a92715d84e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3641c4ec-f761-489e-87f0-26a92715d84e" (UID: "3641c4ec-f761-489e-87f0-26a92715d84e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:37:28.427669 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:37:28.427635    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3641c4ec-f761-489e-87f0-26a92715d84e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3641c4ec-f761-489e-87f0-26a92715d84e" (UID: "3641c4ec-f761-489e-87f0-26a92715d84e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:37:28.465888 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:37:28.465866    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l445j\" (UniqueName: \"kubernetes.io/projected/3641c4ec-f761-489e-87f0-26a92715d84e-kube-api-access-l445j\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 13 19:37:28.465888 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:37:28.465888    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3641c4ec-f761-489e-87f0-26a92715d84e-tmp-dir\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 13 19:37:28.466025 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:37:28.465898    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3641c4ec-f761-489e-87f0-26a92715d84e-tls-certs\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 13 19:37:28.466025 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:37:28.465907    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3641c4ec-f761-489e-87f0-26a92715d84e-kserve-provision-location\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 13 19:37:28.466025 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:37:28.465915    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3641c4ec-f761-489e-87f0-26a92715d84e-home\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 13 19:37:28.466025 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:37:28.465922    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3641c4ec-f761-489e-87f0-26a92715d84e-dshm\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 13 19:37:28.985365 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:37:28.985335    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-9d68b57d4-trssd"
Jul 13 19:37:28.985545 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:37:28.985375    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-9d68b57d4-trssd" event={"ID":"5e36bb45-cdf3-44e8-8b67-8bab1385aa99","Type":"ContainerDied","Data":"3789a29cd31f31dcff9fc7350dc13f95a39e58c1b731c000217806e27a6a214b"}
Jul 13 19:37:28.985545 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:37:28.985443    2565 scope.go:117] "RemoveContainer" containerID="e2a78c498e6f9c98b0c890fd7da7400ca1720a1bdc18bb3c16792ab9a5595e47"
Jul 13 19:37:28.987542 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:37:28.987513    2565 generic.go:356] "Generic (PLEG): container finished" podID="3641c4ec-f761-489e-87f0-26a92715d84e" containerID="b70ebd173b629a5bf6977783af3790bf844d9769aa4aa1286919e4a9ed93bcc6" exitCode=0
Jul 13 19:37:28.987650 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:37:28.987565    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-664bd66db5-cjjg6" event={"ID":"3641c4ec-f761-489e-87f0-26a92715d84e","Type":"ContainerDied","Data":"b70ebd173b629a5bf6977783af3790bf844d9769aa4aa1286919e4a9ed93bcc6"}
Jul 13 19:37:28.987650 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:37:28.987578    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-664bd66db5-cjjg6"
Jul 13 19:37:28.987650 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:37:28.987627    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-664bd66db5-cjjg6" event={"ID":"3641c4ec-f761-489e-87f0-26a92715d84e","Type":"ContainerDied","Data":"db25a18cf277d97aecfb8e1c886b8ff84d7eeb8edfe81eb68b8f046ac4a3f414"}
Jul 13 19:37:28.987809 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:37:28.987647    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mhsdz"
Jul 13 19:37:28.987864 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:37:28.987832    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8gwt"
Jul 13 19:37:28.998154 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:37:28.998100    2565 scope.go:117] "RemoveContainer" containerID="3d995604e7069e8a7ca5bc3441c5b188b8f7ec45ac9e1639c2c7455dbdda7a6b"
Jul 13 19:37:29.006358 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:37:29.006343    2565 scope.go:117] "RemoveContainer" containerID="f1d49422b184040fe76d8441320324d91d54121081c4414cf1ff80cbf4011daa"
Jul 13 19:37:29.014118 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:37:29.014102    2565 scope.go:117] "RemoveContainer" containerID="b97dd6145053085cdf2a73c44e217b8ef3cbe7acc36faaa702e95f6beedabf9e"
Jul 13 19:37:29.015869 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:37:29.015690    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-664bd66db5-cjjg6"]
Jul 13 19:37:29.018482 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:37:29.018459    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-664bd66db5-cjjg6"]
Jul 13 19:37:29.022316 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:37:29.022300    2565 scope.go:117] "RemoveContainer" containerID="b70ebd173b629a5bf6977783af3790bf844d9769aa4aa1286919e4a9ed93bcc6"
Jul 13 19:37:29.029091 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:37:29.029071    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-9d68b57d4-trssd"]
Jul 13 19:37:29.030102 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:37:29.030076    2565 scope.go:117] "RemoveContainer" containerID="f1d49422b184040fe76d8441320324d91d54121081c4414cf1ff80cbf4011daa"
Jul 13 19:37:29.030390 ip-10-0-128-215 kubenswrapper[2565]: E0713 19:37:29.030372    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1d49422b184040fe76d8441320324d91d54121081c4414cf1ff80cbf4011daa\": container with ID starting with f1d49422b184040fe76d8441320324d91d54121081c4414cf1ff80cbf4011daa not found: ID does not exist" containerID="f1d49422b184040fe76d8441320324d91d54121081c4414cf1ff80cbf4011daa"
Jul 13 19:37:29.030443 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:37:29.030399    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1d49422b184040fe76d8441320324d91d54121081c4414cf1ff80cbf4011daa"} err="failed to get container status \"f1d49422b184040fe76d8441320324d91d54121081c4414cf1ff80cbf4011daa\": rpc error: code = NotFound desc = could not find container \"f1d49422b184040fe76d8441320324d91d54121081c4414cf1ff80cbf4011daa\": container with ID starting with f1d49422b184040fe76d8441320324d91d54121081c4414cf1ff80cbf4011daa not found: ID does not exist"
Jul 13 19:37:29.030443 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:37:29.030418    2565 scope.go:117] "RemoveContainer" containerID="b97dd6145053085cdf2a73c44e217b8ef3cbe7acc36faaa702e95f6beedabf9e"
Jul 13 19:37:29.030678 ip-10-0-128-215 kubenswrapper[2565]: E0713 19:37:29.030660    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b97dd6145053085cdf2a73c44e217b8ef3cbe7acc36faaa702e95f6beedabf9e\": container with ID starting with b97dd6145053085cdf2a73c44e217b8ef3cbe7acc36faaa702e95f6beedabf9e not found: ID does not exist" containerID="b97dd6145053085cdf2a73c44e217b8ef3cbe7acc36faaa702e95f6beedabf9e"
Jul 13 19:37:29.030745 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:37:29.030684    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b97dd6145053085cdf2a73c44e217b8ef3cbe7acc36faaa702e95f6beedabf9e"} err="failed to get container status \"b97dd6145053085cdf2a73c44e217b8ef3cbe7acc36faaa702e95f6beedabf9e\": rpc error: code = NotFound desc = could not find container \"b97dd6145053085cdf2a73c44e217b8ef3cbe7acc36faaa702e95f6beedabf9e\": container with ID starting with b97dd6145053085cdf2a73c44e217b8ef3cbe7acc36faaa702e95f6beedabf9e not found: ID does not exist"
Jul 13 19:37:29.030745 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:37:29.030716    2565 scope.go:117] "RemoveContainer" containerID="b70ebd173b629a5bf6977783af3790bf844d9769aa4aa1286919e4a9ed93bcc6"
Jul 13 19:37:29.030967 ip-10-0-128-215 kubenswrapper[2565]: E0713 19:37:29.030950    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b70ebd173b629a5bf6977783af3790bf844d9769aa4aa1286919e4a9ed93bcc6\": container with ID starting with b70ebd173b629a5bf6977783af3790bf844d9769aa4aa1286919e4a9ed93bcc6 not found: ID does not exist" containerID="b70ebd173b629a5bf6977783af3790bf844d9769aa4aa1286919e4a9ed93bcc6"
Jul 13 19:37:29.031006 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:37:29.030973    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b70ebd173b629a5bf6977783af3790bf844d9769aa4aa1286919e4a9ed93bcc6"} err="failed to get container status \"b70ebd173b629a5bf6977783af3790bf844d9769aa4aa1286919e4a9ed93bcc6\": rpc error: code = NotFound desc = could not find container \"b70ebd173b629a5bf6977783af3790bf844d9769aa4aa1286919e4a9ed93bcc6\": container with ID starting with b70ebd173b629a5bf6977783af3790bf844d9769aa4aa1286919e4a9ed93bcc6 not found: ID does not exist"
Jul 13 19:37:29.034327 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:37:29.034307    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-9d68b57d4-trssd"]
Jul 13 19:37:30.048213 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:37:30.048171    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3641c4ec-f761-489e-87f0-26a92715d84e" path="/var/lib/kubelet/pods/3641c4ec-f761-489e-87f0-26a92715d84e/volumes"
Jul 13 19:37:30.048870 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:37:30.048849    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5e36bb45-cdf3-44e8-8b67-8bab1385aa99" path="/var/lib/kubelet/pods/5e36bb45-cdf3-44e8-8b67-8bab1385aa99/volumes"
Jul 13 19:37:52.934984 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:37:52.934949    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5c94bf7bb75w9xs"]
Jul 13 19:37:52.935383 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:37:52.935373    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3641c4ec-f761-489e-87f0-26a92715d84e" containerName="storage-initializer"
Jul 13 19:37:52.935427 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:37:52.935384    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="3641c4ec-f761-489e-87f0-26a92715d84e" containerName="storage-initializer"
Jul 13 19:37:52.935427 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:37:52.935392    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3641c4ec-f761-489e-87f0-26a92715d84e" containerName="main"
Jul 13 19:37:52.935427 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:37:52.935397    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="3641c4ec-f761-489e-87f0-26a92715d84e" containerName="main"
Jul 13 19:37:52.935427 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:37:52.935410    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e36bb45-cdf3-44e8-8b67-8bab1385aa99" containerName="storage-initializer"
Jul 13 19:37:52.935427 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:37:52.935417    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e36bb45-cdf3-44e8-8b67-8bab1385aa99" containerName="storage-initializer"
Jul 13 19:37:52.935427 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:37:52.935427    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e36bb45-cdf3-44e8-8b67-8bab1385aa99" containerName="main"
Jul 13 19:37:52.935601 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:37:52.935432    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e36bb45-cdf3-44e8-8b67-8bab1385aa99" containerName="main"
Jul 13 19:37:52.935601 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:37:52.935440    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3641c4ec-f761-489e-87f0-26a92715d84e" containerName="llm-d-routing-sidecar"
Jul 13 19:37:52.935601 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:37:52.935445    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="3641c4ec-f761-489e-87f0-26a92715d84e" containerName="llm-d-routing-sidecar"
Jul 13 19:37:52.935601 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:37:52.935501    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="5e36bb45-cdf3-44e8-8b67-8bab1385aa99" containerName="main"
Jul 13 19:37:52.935601 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:37:52.935509    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="3641c4ec-f761-489e-87f0-26a92715d84e" containerName="llm-d-routing-sidecar"
Jul 13 19:37:52.935601 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:37:52.935515    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="3641c4ec-f761-489e-87f0-26a92715d84e" containerName="main"
Jul 13 19:37:52.938663 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:37:52.938645    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5c94bf7bb75w9xs"
Jul 13 19:37:52.941123 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:37:52.941104    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 13 19:37:52.941234 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:37:52.941215    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvd10b93c8eba12f87c3a350a5cae2ee0c-kserve-self-signed-certs\""
Jul 13 19:37:52.949618 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:37:52.949598    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5c94bf7bb75w9xs"]
Jul 13 19:37:52.987594 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:37:52.987567    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3000bcee-b26f-4faf-816a-05547d90a554-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-5c94bf7bb75w9xs\" (UID: \"3000bcee-b26f-4faf-816a-05547d90a554\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5c94bf7bb75w9xs"
Jul 13 19:37:52.987730 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:37:52.987600    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3000bcee-b26f-4faf-816a-05547d90a554-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-5c94bf7bb75w9xs\" (UID: \"3000bcee-b26f-4faf-816a-05547d90a554\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5c94bf7bb75w9xs"
Jul 13 19:37:52.987730 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:37:52.987690    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpkpk\" (UniqueName: \"kubernetes.io/projected/3000bcee-b26f-4faf-816a-05547d90a554-kube-api-access-qpkpk\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-5c94bf7bb75w9xs\" (UID: \"3000bcee-b26f-4faf-816a-05547d90a554\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5c94bf7bb75w9xs"
Jul 13 19:37:52.987809 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:37:52.987764    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3000bcee-b26f-4faf-816a-05547d90a554-dshm\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-5c94bf7bb75w9xs\" (UID: \"3000bcee-b26f-4faf-816a-05547d90a554\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5c94bf7bb75w9xs"
Jul 13 19:37:52.987865 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:37:52.987849    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3000bcee-b26f-4faf-816a-05547d90a554-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-5c94bf7bb75w9xs\" (UID: \"3000bcee-b26f-4faf-816a-05547d90a554\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5c94bf7bb75w9xs"
Jul 13 19:37:52.987901 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:37:52.987874    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3000bcee-b26f-4faf-816a-05547d90a554-home\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-5c94bf7bb75w9xs\" (UID: \"3000bcee-b26f-4faf-816a-05547d90a554\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5c94bf7bb75w9xs"
Jul 13 19:37:52.987936 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:37:52.987901    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3000bcee-b26f-4faf-816a-05547d90a554-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-5c94bf7bb75w9xs\" (UID: \"3000bcee-b26f-4faf-816a-05547d90a554\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5c94bf7bb75w9xs"
Jul 13 19:37:53.088304 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:37:53.088278    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3000bcee-b26f-4faf-816a-05547d90a554-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-5c94bf7bb75w9xs\" (UID: \"3000bcee-b26f-4faf-816a-05547d90a554\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5c94bf7bb75w9xs"
Jul 13 19:37:53.088440 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:37:53.088309    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3000bcee-b26f-4faf-816a-05547d90a554-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-5c94bf7bb75w9xs\" (UID: \"3000bcee-b26f-4faf-816a-05547d90a554\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5c94bf7bb75w9xs"
Jul 13 19:37:53.088511 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:37:53.088454    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpkpk\" (UniqueName: \"kubernetes.io/projected/3000bcee-b26f-4faf-816a-05547d90a554-kube-api-access-qpkpk\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-5c94bf7bb75w9xs\" (UID: \"3000bcee-b26f-4faf-816a-05547d90a554\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5c94bf7bb75w9xs"
Jul 13 19:37:53.088511 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:37:53.088493    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3000bcee-b26f-4faf-816a-05547d90a554-dshm\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-5c94bf7bb75w9xs\" (UID: \"3000bcee-b26f-4faf-816a-05547d90a554\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5c94bf7bb75w9xs"
Jul 13 19:37:53.088685 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:37:53.088661    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3000bcee-b26f-4faf-816a-05547d90a554-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-5c94bf7bb75w9xs\" (UID: \"3000bcee-b26f-4faf-816a-05547d90a554\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5c94bf7bb75w9xs"
Jul 13 19:37:53.088887 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:37:53.088731    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3000bcee-b26f-4faf-816a-05547d90a554-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-5c94bf7bb75w9xs\" (UID: \"3000bcee-b26f-4faf-816a-05547d90a554\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5c94bf7bb75w9xs"
Jul 13 19:37:53.088991 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:37:53.088969    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3000bcee-b26f-4faf-816a-05547d90a554-home\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-5c94bf7bb75w9xs\" (UID: \"3000bcee-b26f-4faf-816a-05547d90a554\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5c94bf7bb75w9xs"
Jul 13 19:37:53.089264 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:37:53.088994    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3000bcee-b26f-4faf-816a-05547d90a554-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-5c94bf7bb75w9xs\" (UID: \"3000bcee-b26f-4faf-816a-05547d90a554\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5c94bf7bb75w9xs"
Jul 13 19:37:53.089264 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:37:53.089003    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3000bcee-b26f-4faf-816a-05547d90a554-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-5c94bf7bb75w9xs\" (UID: \"3000bcee-b26f-4faf-816a-05547d90a554\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5c94bf7bb75w9xs"
Jul 13 19:37:53.089264 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:37:53.089254    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3000bcee-b26f-4faf-816a-05547d90a554-home\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-5c94bf7bb75w9xs\" (UID: \"3000bcee-b26f-4faf-816a-05547d90a554\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5c94bf7bb75w9xs"
Jul 13 19:37:53.089473 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:37:53.089449    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3000bcee-b26f-4faf-816a-05547d90a554-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-5c94bf7bb75w9xs\" (UID: \"3000bcee-b26f-4faf-816a-05547d90a554\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5c94bf7bb75w9xs"
Jul 13 19:37:53.090933 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:37:53.090902    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3000bcee-b26f-4faf-816a-05547d90a554-dshm\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-5c94bf7bb75w9xs\" (UID: \"3000bcee-b26f-4faf-816a-05547d90a554\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5c94bf7bb75w9xs"
Jul 13 19:37:53.091016 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:37:53.091001    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3000bcee-b26f-4faf-816a-05547d90a554-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-5c94bf7bb75w9xs\" (UID: \"3000bcee-b26f-4faf-816a-05547d90a554\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5c94bf7bb75w9xs"
Jul 13 19:37:53.097368 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:37:53.097350    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpkpk\" (UniqueName: \"kubernetes.io/projected/3000bcee-b26f-4faf-816a-05547d90a554-kube-api-access-qpkpk\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-5c94bf7bb75w9xs\" (UID: \"3000bcee-b26f-4faf-816a-05547d90a554\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5c94bf7bb75w9xs"
Jul 13 19:37:53.249827 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:37:53.249755    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5c94bf7bb75w9xs"
Jul 13 19:37:53.386887 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:37:53.386864    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5c94bf7bb75w9xs"]
Jul 13 19:37:53.388885 ip-10-0-128-215 kubenswrapper[2565]: W0713 19:37:53.388859    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3000bcee_b26f_4faf_816a_05547d90a554.slice/crio-ef9efefb599b8a3cfc099f651589ed32bb99e79a1ae5e4528f8bc1a6570fca3e WatchSource:0}: Error finding container ef9efefb599b8a3cfc099f651589ed32bb99e79a1ae5e4528f8bc1a6570fca3e: Status 404 returned error can't find the container with id ef9efefb599b8a3cfc099f651589ed32bb99e79a1ae5e4528f8bc1a6570fca3e
Jul 13 19:37:54.107279 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:37:54.107244    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5c94bf7bb75w9xs" event={"ID":"3000bcee-b26f-4faf-816a-05547d90a554","Type":"ContainerStarted","Data":"226734280a3059d9c0abc00fa79625d3fcc087ba26145b97be4b0b2f6fe3a794"}
Jul 13 19:37:54.107616 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:37:54.107301    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5c94bf7bb75w9xs" event={"ID":"3000bcee-b26f-4faf-816a-05547d90a554","Type":"ContainerStarted","Data":"ef9efefb599b8a3cfc099f651589ed32bb99e79a1ae5e4528f8bc1a6570fca3e"}
Jul 13 19:37:57.130143 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:37:57.130107    2565 generic.go:356] "Generic (PLEG): container finished" podID="3000bcee-b26f-4faf-816a-05547d90a554" containerID="226734280a3059d9c0abc00fa79625d3fcc087ba26145b97be4b0b2f6fe3a794" exitCode=0
Jul 13 19:37:57.130619 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:37:57.130194    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5c94bf7bb75w9xs" event={"ID":"3000bcee-b26f-4faf-816a-05547d90a554","Type":"ContainerDied","Data":"226734280a3059d9c0abc00fa79625d3fcc087ba26145b97be4b0b2f6fe3a794"}
Jul 13 19:37:58.136961 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:37:58.136921    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5c94bf7bb75w9xs" event={"ID":"3000bcee-b26f-4faf-816a-05547d90a554","Type":"ContainerStarted","Data":"bf2636b94a07b2c237da68e91e1d1a983b7e0bd72488ecc4586c8e0047119fd4"}
Jul 13 19:37:58.161825 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:37:58.161778    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5c94bf7bb75w9xs" podStartSLOduration=3.208484885 podStartE2EDuration="6.161761609s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.953276724s" isStatefulPod=false podCreationTimestamp="2026-07-13 19:37:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 19:37:58.158777033 +0000 UTC m=+2864.683910495" watchObservedRunningTime="2026-07-13 19:37:58.161761609 +0000 UTC m=+2864.686895023"
Jul 13 19:37:58.250787 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:37:58.250757    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5c94bf7bb75w9xs"
Jul 13 19:38:03.250280 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:38:03.250241    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5c94bf7bb75w9xs"
Jul 13 19:38:03.252083 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:38:03.252052    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5c94bf7bb75w9xs" podUID="3000bcee-b26f-4faf-816a-05547d90a554" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": dial tcp 10.134.0.61:8000: connect: connection refused"
Jul 13 19:38:13.251059 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:38:13.250968    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5c94bf7bb75w9xs" podUID="3000bcee-b26f-4faf-816a-05547d90a554" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": dial tcp 10.134.0.61:8000: connect: connection refused"
Jul 13 19:38:23.250650 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:38:23.250608    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5c94bf7bb75w9xs" podUID="3000bcee-b26f-4faf-816a-05547d90a554" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": dial tcp 10.134.0.61:8000: connect: connection refused"
Jul 13 19:38:27.655563 ip-10-0-128-215 kubenswrapper[2565]: E0713 19:38:27.655515    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dmww7f" podUID="634b78c4-751c-4880-9f6c-1ea69b737987"
Jul 13 19:38:28.297145 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:38:28.297114    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dmww7f"
Jul 13 19:38:33.250136 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:38:33.250096    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5c94bf7bb75w9xs" podUID="3000bcee-b26f-4faf-816a-05547d90a554" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": dial tcp 10.134.0.61:8000: connect: connection refused"
Jul 13 19:38:43.250328 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:38:43.250283    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5c94bf7bb75w9xs" podUID="3000bcee-b26f-4faf-816a-05547d90a554" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": dial tcp 10.134.0.61:8000: connect: connection refused"
Jul 13 19:38:53.251091 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:38:53.251052    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5c94bf7bb75w9xs" podUID="3000bcee-b26f-4faf-816a-05547d90a554" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": dial tcp 10.134.0.61:8000: connect: connection refused"
Jul 13 19:39:03.250603 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:39:03.250556    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5c94bf7bb75w9xs" podUID="3000bcee-b26f-4faf-816a-05547d90a554" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": dial tcp 10.134.0.61:8000: connect: connection refused"
Jul 13 19:39:13.250746 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:39:13.250676    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5c94bf7bb75w9xs" podUID="3000bcee-b26f-4faf-816a-05547d90a554" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": dial tcp 10.134.0.61:8000: connect: connection refused"
Jul 13 19:39:23.250109 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:39:23.250060    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5c94bf7bb75w9xs" podUID="3000bcee-b26f-4faf-816a-05547d90a554" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": dial tcp 10.134.0.61:8000: connect: connection refused"
Jul 13 19:39:31.989656 ip-10-0-128-215 kubenswrapper[2565]: E0713 19:39:31.989596    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8gwt" podUID="74e7b5d4-9d6e-4736-98c3-09d48ed6dec2"
Jul 13 19:39:31.989656 ip-10-0-128-215 kubenswrapper[2565]: E0713 19:39:31.989635    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mhsdz" podUID="762ad0f9-19a6-4a36-b4eb-a50f7291ab13"
Jul 13 19:39:32.622239 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:39:32.622203    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8gwt"
Jul 13 19:39:32.622476 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:39:32.622204    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mhsdz"
Jul 13 19:39:33.251042 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:39:33.250979    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5c94bf7bb75w9xs" podUID="3000bcee-b26f-4faf-816a-05547d90a554" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": dial tcp 10.134.0.61:8000: connect: connection refused"
Jul 13 19:39:43.250595 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:39:43.250503    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5c94bf7bb75w9xs" podUID="3000bcee-b26f-4faf-816a-05547d90a554" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": dial tcp 10.134.0.61:8000: connect: connection refused"
Jul 13 19:39:53.260370 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:39:53.260338    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5c94bf7bb75w9xs"
Jul 13 19:39:53.268161 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:39:53.268140    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5c94bf7bb75w9xs"
Jul 13 19:40:14.181871 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:40:14.181844    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2bscc_2f0655bd-4a09-4314-8c1b-e30191eaec0e/ovn-acl-logging/0.log"
Jul 13 19:40:14.185271 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:40:14.185246    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2bscc_2f0655bd-4a09-4314-8c1b-e30191eaec0e/ovn-acl-logging/0.log"
Jul 13 19:40:31.298069 ip-10-0-128-215 kubenswrapper[2565]: E0713 19:40:31.298029    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dmww7f" podUID="634b78c4-751c-4880-9f6c-1ea69b737987"
Jul 13 19:40:31.920076 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:40:31.920046    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dmww7f"
Jul 13 19:40:46.652159 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:40:46.652128    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5c94bf7bb75w9xs"]
Jul 13 19:41:01.675869 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:41:01.675821    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5c94bf7bb75w9xs" podUID="3000bcee-b26f-4faf-816a-05547d90a554" containerName="main" containerID="cri-o://bf2636b94a07b2c237da68e91e1d1a983b7e0bd72488ecc4586c8e0047119fd4" gracePeriod=45
Jul 13 19:41:02.261076 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:41:02.261034    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5c94bf7bb75w9xs" podUID="3000bcee-b26f-4faf-816a-05547d90a554" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": dial tcp 10.134.0.61:8000: connect: connection refused"
Jul 13 19:41:02.435645 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:41:02.435570    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5c94bf7bb75w9xs"
Jul 13 19:41:02.484934 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:41:02.484898    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3000bcee-b26f-4faf-816a-05547d90a554-dshm\") pod \"3000bcee-b26f-4faf-816a-05547d90a554\" (UID: \"3000bcee-b26f-4faf-816a-05547d90a554\") "
Jul 13 19:41:02.485116 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:41:02.484959    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3000bcee-b26f-4faf-816a-05547d90a554-kserve-provision-location\") pod \"3000bcee-b26f-4faf-816a-05547d90a554\" (UID: \"3000bcee-b26f-4faf-816a-05547d90a554\") "
Jul 13 19:41:02.485116 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:41:02.484986    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3000bcee-b26f-4faf-816a-05547d90a554-home\") pod \"3000bcee-b26f-4faf-816a-05547d90a554\" (UID: \"3000bcee-b26f-4faf-816a-05547d90a554\") "
Jul 13 19:41:02.485116 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:41:02.485015    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qpkpk\" (UniqueName: \"kubernetes.io/projected/3000bcee-b26f-4faf-816a-05547d90a554-kube-api-access-qpkpk\") pod \"3000bcee-b26f-4faf-816a-05547d90a554\" (UID: \"3000bcee-b26f-4faf-816a-05547d90a554\") "
Jul 13 19:41:02.485116 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:41:02.485030    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3000bcee-b26f-4faf-816a-05547d90a554-model-cache\") pod \"3000bcee-b26f-4faf-816a-05547d90a554\" (UID: \"3000bcee-b26f-4faf-816a-05547d90a554\") "
Jul 13 19:41:02.485531 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:41:02.485493    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3000bcee-b26f-4faf-816a-05547d90a554-model-cache" (OuterVolumeSpecName: "model-cache") pod "3000bcee-b26f-4faf-816a-05547d90a554" (UID: "3000bcee-b26f-4faf-816a-05547d90a554"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:41:02.485904 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:41:02.485879    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3000bcee-b26f-4faf-816a-05547d90a554-home" (OuterVolumeSpecName: "home") pod "3000bcee-b26f-4faf-816a-05547d90a554" (UID: "3000bcee-b26f-4faf-816a-05547d90a554"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:41:02.487360 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:41:02.487338    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3000bcee-b26f-4faf-816a-05547d90a554-dshm" (OuterVolumeSpecName: "dshm") pod "3000bcee-b26f-4faf-816a-05547d90a554" (UID: "3000bcee-b26f-4faf-816a-05547d90a554"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:41:02.487432 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:41:02.487358    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3000bcee-b26f-4faf-816a-05547d90a554-kube-api-access-qpkpk" (OuterVolumeSpecName: "kube-api-access-qpkpk") pod "3000bcee-b26f-4faf-816a-05547d90a554" (UID: "3000bcee-b26f-4faf-816a-05547d90a554"). InnerVolumeSpecName "kube-api-access-qpkpk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 19:41:02.552486 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:41:02.552444    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3000bcee-b26f-4faf-816a-05547d90a554-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3000bcee-b26f-4faf-816a-05547d90a554" (UID: "3000bcee-b26f-4faf-816a-05547d90a554"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:41:02.586221 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:41:02.586181    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3000bcee-b26f-4faf-816a-05547d90a554-tmp-dir\") pod \"3000bcee-b26f-4faf-816a-05547d90a554\" (UID: \"3000bcee-b26f-4faf-816a-05547d90a554\") "
Jul 13 19:41:02.586221 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:41:02.586223    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3000bcee-b26f-4faf-816a-05547d90a554-tls-certs\") pod \"3000bcee-b26f-4faf-816a-05547d90a554\" (UID: \"3000bcee-b26f-4faf-816a-05547d90a554\") "
Jul 13 19:41:02.586480 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:41:02.586468    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3000bcee-b26f-4faf-816a-05547d90a554-dshm\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 13 19:41:02.586522 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:41:02.586482    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3000bcee-b26f-4faf-816a-05547d90a554-kserve-provision-location\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 13 19:41:02.586522 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:41:02.586494    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3000bcee-b26f-4faf-816a-05547d90a554-home\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 13 19:41:02.586522 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:41:02.586502    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qpkpk\" (UniqueName: \"kubernetes.io/projected/3000bcee-b26f-4faf-816a-05547d90a554-kube-api-access-qpkpk\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 13 19:41:02.586522 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:41:02.586510    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3000bcee-b26f-4faf-816a-05547d90a554-model-cache\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 13 19:41:02.588520 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:41:02.588477    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3000bcee-b26f-4faf-816a-05547d90a554-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3000bcee-b26f-4faf-816a-05547d90a554" (UID: "3000bcee-b26f-4faf-816a-05547d90a554"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 19:41:02.597551 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:41:02.597510    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3000bcee-b26f-4faf-816a-05547d90a554-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3000bcee-b26f-4faf-816a-05547d90a554" (UID: "3000bcee-b26f-4faf-816a-05547d90a554"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:41:02.687151 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:41:02.687059    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3000bcee-b26f-4faf-816a-05547d90a554-tmp-dir\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 13 19:41:02.687151 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:41:02.687092    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3000bcee-b26f-4faf-816a-05547d90a554-tls-certs\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 13 19:41:03.084907 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:41:03.084873    2565 generic.go:356] "Generic (PLEG): container finished" podID="3000bcee-b26f-4faf-816a-05547d90a554" containerID="bf2636b94a07b2c237da68e91e1d1a983b7e0bd72488ecc4586c8e0047119fd4" exitCode=0
Jul 13 19:41:03.085083 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:41:03.084939    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5c94bf7bb75w9xs"
Jul 13 19:41:03.085083 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:41:03.084968    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5c94bf7bb75w9xs" event={"ID":"3000bcee-b26f-4faf-816a-05547d90a554","Type":"ContainerDied","Data":"bf2636b94a07b2c237da68e91e1d1a983b7e0bd72488ecc4586c8e0047119fd4"}
Jul 13 19:41:03.085083 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:41:03.085037    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5c94bf7bb75w9xs" event={"ID":"3000bcee-b26f-4faf-816a-05547d90a554","Type":"ContainerDied","Data":"ef9efefb599b8a3cfc099f651589ed32bb99e79a1ae5e4528f8bc1a6570fca3e"}
Jul 13 19:41:03.085083 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:41:03.085066    2565 scope.go:117] "RemoveContainer" containerID="bf2636b94a07b2c237da68e91e1d1a983b7e0bd72488ecc4586c8e0047119fd4"
Jul 13 19:41:03.094238 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:41:03.094222    2565 scope.go:117] "RemoveContainer" containerID="226734280a3059d9c0abc00fa79625d3fcc087ba26145b97be4b0b2f6fe3a794"
Jul 13 19:41:03.102439 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:41:03.102417    2565 scope.go:117] "RemoveContainer" containerID="bf2636b94a07b2c237da68e91e1d1a983b7e0bd72488ecc4586c8e0047119fd4"
Jul 13 19:41:03.102712 ip-10-0-128-215 kubenswrapper[2565]: E0713 19:41:03.102680    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf2636b94a07b2c237da68e91e1d1a983b7e0bd72488ecc4586c8e0047119fd4\": container with ID starting with bf2636b94a07b2c237da68e91e1d1a983b7e0bd72488ecc4586c8e0047119fd4 not found: ID does not exist" containerID="bf2636b94a07b2c237da68e91e1d1a983b7e0bd72488ecc4586c8e0047119fd4"
Jul 13 19:41:03.102771 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:41:03.102725    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf2636b94a07b2c237da68e91e1d1a983b7e0bd72488ecc4586c8e0047119fd4"} err="failed to get container status \"bf2636b94a07b2c237da68e91e1d1a983b7e0bd72488ecc4586c8e0047119fd4\": rpc error: code = NotFound desc = could not find container \"bf2636b94a07b2c237da68e91e1d1a983b7e0bd72488ecc4586c8e0047119fd4\": container with ID starting with bf2636b94a07b2c237da68e91e1d1a983b7e0bd72488ecc4586c8e0047119fd4 not found: ID does not exist"
Jul 13 19:41:03.102771 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:41:03.102745    2565 scope.go:117] "RemoveContainer" containerID="226734280a3059d9c0abc00fa79625d3fcc087ba26145b97be4b0b2f6fe3a794"
Jul 13 19:41:03.102961 ip-10-0-128-215 kubenswrapper[2565]: E0713 19:41:03.102946    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"226734280a3059d9c0abc00fa79625d3fcc087ba26145b97be4b0b2f6fe3a794\": container with ID starting with 226734280a3059d9c0abc00fa79625d3fcc087ba26145b97be4b0b2f6fe3a794 not found: ID does not exist" containerID="226734280a3059d9c0abc00fa79625d3fcc087ba26145b97be4b0b2f6fe3a794"
Jul 13 19:41:03.103002 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:41:03.102967    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"226734280a3059d9c0abc00fa79625d3fcc087ba26145b97be4b0b2f6fe3a794"} err="failed to get container status \"226734280a3059d9c0abc00fa79625d3fcc087ba26145b97be4b0b2f6fe3a794\": rpc error: code = NotFound desc = could not find container \"226734280a3059d9c0abc00fa79625d3fcc087ba26145b97be4b0b2f6fe3a794\": container with ID starting with 226734280a3059d9c0abc00fa79625d3fcc087ba26145b97be4b0b2f6fe3a794 not found: ID does not exist"
Jul 13 19:41:03.110499 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:41:03.110469    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5c94bf7bb75w9xs"]
Jul 13 19:41:03.114837 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:41:03.114808    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-5c94bf7bb75w9xs"]
Jul 13 19:41:04.048866 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:41:04.048781    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3000bcee-b26f-4faf-816a-05547d90a554" path="/var/lib/kubelet/pods/3000bcee-b26f-4faf-816a-05547d90a554/volumes"
Jul 13 19:41:30.489802 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:41:30.489754    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Jul 13 19:41:30.490467 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:41:30.490268    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3000bcee-b26f-4faf-816a-05547d90a554" containerName="storage-initializer"
Jul 13 19:41:30.490467 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:41:30.490288    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="3000bcee-b26f-4faf-816a-05547d90a554" containerName="storage-initializer"
Jul 13 19:41:30.490467 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:41:30.490299    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3000bcee-b26f-4faf-816a-05547d90a554" containerName="main"
Jul 13 19:41:30.490467 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:41:30.490308    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="3000bcee-b26f-4faf-816a-05547d90a554" containerName="main"
Jul 13 19:41:30.490467 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:41:30.490410    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="3000bcee-b26f-4faf-816a-05547d90a554" containerName="main"
Jul 13 19:41:30.493728 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:41:30.493691    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 13 19:41:30.496449 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:41:30.496425    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 13 19:41:30.496579 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:41:30.496485    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-fb-opt-125m-route-dc21cb-57cf7736-dockercfg-wwgss\""
Jul 13 19:41:30.497409 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:41:30.497392    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs\""
Jul 13 19:41:30.514958 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:41:30.514932    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Jul 13 19:41:30.560439 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:41:30.560404    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a46fc802-362c-4cd4-9d04-dd1ae4aa348f-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"a46fc802-362c-4cd4-9d04-dd1ae4aa348f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 13 19:41:30.560636 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:41:30.560446    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdq2f\" (UniqueName: \"kubernetes.io/projected/a46fc802-362c-4cd4-9d04-dd1ae4aa348f-kube-api-access-mdq2f\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"a46fc802-362c-4cd4-9d04-dd1ae4aa348f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 13 19:41:30.560636 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:41:30.560476    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a46fc802-362c-4cd4-9d04-dd1ae4aa348f-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"a46fc802-362c-4cd4-9d04-dd1ae4aa348f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 13 19:41:30.560636 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:41:30.560531    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a46fc802-362c-4cd4-9d04-dd1ae4aa348f-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"a46fc802-362c-4cd4-9d04-dd1ae4aa348f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 13 19:41:30.560636 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:41:30.560555    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a46fc802-362c-4cd4-9d04-dd1ae4aa348f-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"a46fc802-362c-4cd4-9d04-dd1ae4aa348f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 13 19:41:30.560636 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:41:30.560614    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a46fc802-362c-4cd4-9d04-dd1ae4aa348f-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"a46fc802-362c-4cd4-9d04-dd1ae4aa348f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 13 19:41:30.560861 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:41:30.560679    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a46fc802-362c-4cd4-9d04-dd1ae4aa348f-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"a46fc802-362c-4cd4-9d04-dd1ae4aa348f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 13 19:41:30.661517 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:41:30.661471    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a46fc802-362c-4cd4-9d04-dd1ae4aa348f-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"a46fc802-362c-4cd4-9d04-dd1ae4aa348f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 13 19:41:30.661764 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:41:30.661526    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a46fc802-362c-4cd4-9d04-dd1ae4aa348f-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"a46fc802-362c-4cd4-9d04-dd1ae4aa348f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 13 19:41:30.661764 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:41:30.661618    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a46fc802-362c-4cd4-9d04-dd1ae4aa348f-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"a46fc802-362c-4cd4-9d04-dd1ae4aa348f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 13 19:41:30.661764 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:41:30.661646    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdq2f\" (UniqueName: \"kubernetes.io/projected/a46fc802-362c-4cd4-9d04-dd1ae4aa348f-kube-api-access-mdq2f\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"a46fc802-362c-4cd4-9d04-dd1ae4aa348f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 13 19:41:30.661764 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:41:30.661676    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a46fc802-362c-4cd4-9d04-dd1ae4aa348f-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"a46fc802-362c-4cd4-9d04-dd1ae4aa348f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 13 19:41:30.661764 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:41:30.661747    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a46fc802-362c-4cd4-9d04-dd1ae4aa348f-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"a46fc802-362c-4cd4-9d04-dd1ae4aa348f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 13 19:41:30.662059 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:41:30.661776    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a46fc802-362c-4cd4-9d04-dd1ae4aa348f-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"a46fc802-362c-4cd4-9d04-dd1ae4aa348f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 13 19:41:30.662059 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:41:30.661995    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a46fc802-362c-4cd4-9d04-dd1ae4aa348f-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"a46fc802-362c-4cd4-9d04-dd1ae4aa348f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 13 19:41:30.662177 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:41:30.662094    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a46fc802-362c-4cd4-9d04-dd1ae4aa348f-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"a46fc802-362c-4cd4-9d04-dd1ae4aa348f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 13 19:41:30.662177 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:41:30.662153    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a46fc802-362c-4cd4-9d04-dd1ae4aa348f-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"a46fc802-362c-4cd4-9d04-dd1ae4aa348f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 13 19:41:30.662264 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:41:30.662176    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a46fc802-362c-4cd4-9d04-dd1ae4aa348f-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"a46fc802-362c-4cd4-9d04-dd1ae4aa348f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 13 19:41:30.664551 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:41:30.664529    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a46fc802-362c-4cd4-9d04-dd1ae4aa348f-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"a46fc802-362c-4cd4-9d04-dd1ae4aa348f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 13 19:41:30.664697 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:41:30.664680    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a46fc802-362c-4cd4-9d04-dd1ae4aa348f-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"a46fc802-362c-4cd4-9d04-dd1ae4aa348f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 13 19:41:30.670569 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:41:30.670549    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdq2f\" (UniqueName: \"kubernetes.io/projected/a46fc802-362c-4cd4-9d04-dd1ae4aa348f-kube-api-access-mdq2f\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"a46fc802-362c-4cd4-9d04-dd1ae4aa348f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 13 19:41:30.804801 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:41:30.804765    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 13 19:41:30.946617 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:41:30.946592    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Jul 13 19:41:30.948564 ip-10-0-128-215 kubenswrapper[2565]: W0713 19:41:30.948535    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda46fc802_362c_4cd4_9d04_dd1ae4aa348f.slice/crio-da107e502c6db33b45e1fe0095ed93bbca199eaaa5d5c1e49129a5be10a87d6c WatchSource:0}: Error finding container da107e502c6db33b45e1fe0095ed93bbca199eaaa5d5c1e49129a5be10a87d6c: Status 404 returned error can't find the container with id da107e502c6db33b45e1fe0095ed93bbca199eaaa5d5c1e49129a5be10a87d6c
Jul 13 19:41:31.223408 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:41:31.223307    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"a46fc802-362c-4cd4-9d04-dd1ae4aa348f","Type":"ContainerStarted","Data":"3753f29fa3e4ce934c7ec925c95652880b00ba6427ee41ff840cabe2bb6e6e1b"}
Jul 13 19:41:31.223408 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:41:31.223364    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"a46fc802-362c-4cd4-9d04-dd1ae4aa348f","Type":"ContainerStarted","Data":"da107e502c6db33b45e1fe0095ed93bbca199eaaa5d5c1e49129a5be10a87d6c"}
Jul 13 19:41:34.244739 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:41:34.244674    2565 generic.go:356] "Generic (PLEG): container finished" podID="a46fc802-362c-4cd4-9d04-dd1ae4aa348f" containerID="3753f29fa3e4ce934c7ec925c95652880b00ba6427ee41ff840cabe2bb6e6e1b" exitCode=0
Jul 13 19:41:34.245241 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:41:34.244738    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"a46fc802-362c-4cd4-9d04-dd1ae4aa348f","Type":"ContainerDied","Data":"3753f29fa3e4ce934c7ec925c95652880b00ba6427ee41ff840cabe2bb6e6e1b"}
Jul 13 19:41:35.253947 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:41:35.253908    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"a46fc802-362c-4cd4-9d04-dd1ae4aa348f","Type":"ContainerStarted","Data":"b05922c0285cbc990c2f3d078de1c236fe99964dfb0c2e7c9fb0f011df535f30"}
Jul 13 19:41:35.277115 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:41:35.277055    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podStartSLOduration=2.176800614 podStartE2EDuration="5.277038454s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.10023784s" isStatefulPod=false podCreationTimestamp="2026-07-13 19:41:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 19:41:35.275839073 +0000 UTC m=+3081.800972504" watchObservedRunningTime="2026-07-13 19:41:35.277038454 +0000 UTC m=+3081.802171870"
Jul 13 19:41:35.623180 ip-10-0-128-215 kubenswrapper[2565]: E0713 19:41:35.623131    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8gwt" podUID="74e7b5d4-9d6e-4736-98c3-09d48ed6dec2"
Jul 13 19:41:35.623344 ip-10-0-128-215 kubenswrapper[2565]: E0713 19:41:35.623210    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mhsdz" podUID="762ad0f9-19a6-4a36-b4eb-a50f7291ab13"
Jul 13 19:41:35.805071 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:41:35.805027    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 13 19:41:36.259264 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:41:36.259230    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8gwt"
Jul 13 19:41:36.259264 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:41:36.259230    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mhsdz"
Jul 13 19:41:40.805478 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:41:40.805440    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 13 19:41:40.807043 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:41:40.807015    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="a46fc802-362c-4cd4-9d04-dd1ae4aa348f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": dial tcp 10.134.0.62:8000: connect: connection refused"
Jul 13 19:41:50.805490 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:41:50.805444    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="a46fc802-362c-4cd4-9d04-dd1ae4aa348f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": dial tcp 10.134.0.62:8000: connect: connection refused"
Jul 13 19:42:00.805422 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:42:00.805375    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="a46fc802-362c-4cd4-9d04-dd1ae4aa348f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": dial tcp 10.134.0.62:8000: connect: connection refused"
Jul 13 19:42:10.805951 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:42:10.805894    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="a46fc802-362c-4cd4-9d04-dd1ae4aa348f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": dial tcp 10.134.0.62:8000: connect: connection refused"
Jul 13 19:42:20.806096 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:42:20.806038    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="a46fc802-362c-4cd4-9d04-dd1ae4aa348f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": dial tcp 10.134.0.62:8000: connect: connection refused"
Jul 13 19:42:30.805586 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:42:30.805533    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="a46fc802-362c-4cd4-9d04-dd1ae4aa348f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": dial tcp 10.134.0.62:8000: connect: connection refused"
Jul 13 19:42:34.921771 ip-10-0-128-215 kubenswrapper[2565]: E0713 19:42:34.921652    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dmww7f" podUID="634b78c4-751c-4880-9f6c-1ea69b737987"
Jul 13 19:42:35.561997 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:42:35.561965    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dmww7f"
Jul 13 19:42:40.806010 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:42:40.805967    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="a46fc802-362c-4cd4-9d04-dd1ae4aa348f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": dial tcp 10.134.0.62:8000: connect: connection refused"
Jul 13 19:42:50.806058 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:42:50.806019    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="a46fc802-362c-4cd4-9d04-dd1ae4aa348f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": dial tcp 10.134.0.62:8000: connect: connection refused"
Jul 13 19:43:00.805484 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:43:00.805440    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="a46fc802-362c-4cd4-9d04-dd1ae4aa348f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": dial tcp 10.134.0.62:8000: connect: connection refused"
Jul 13 19:43:10.815734 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:43:10.815668    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 13 19:43:10.823502 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:43:10.823474    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 13 19:43:39.260104 ip-10-0-128-215 kubenswrapper[2565]: E0713 19:43:39.260060    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mhsdz" podUID="762ad0f9-19a6-4a36-b4eb-a50f7291ab13"
Jul 13 19:43:39.260104 ip-10-0-128-215 kubenswrapper[2565]: E0713 19:43:39.260063    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8gwt" podUID="74e7b5d4-9d6e-4736-98c3-09d48ed6dec2"
Jul 13 19:43:39.890573 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:43:39.890539    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8gwt"
Jul 13 19:43:39.890780 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:43:39.890539    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mhsdz"
Jul 13 19:44:38.563238 ip-10-0-128-215 kubenswrapper[2565]: E0713 19:44:38.563145    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dmww7f" podUID="634b78c4-751c-4880-9f6c-1ea69b737987"
Jul 13 19:44:39.191929 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:44:39.191897    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dmww7f"
Jul 13 19:45:14.212940 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:45:14.212908    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2bscc_2f0655bd-4a09-4314-8c1b-e30191eaec0e/ovn-acl-logging/0.log"
Jul 13 19:45:14.219543 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:45:14.219517    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2bscc_2f0655bd-4a09-4314-8c1b-e30191eaec0e/ovn-acl-logging/0.log"
Jul 13 19:45:42.891314 ip-10-0-128-215 kubenswrapper[2565]: E0713 19:45:42.891208    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8gwt" podUID="74e7b5d4-9d6e-4736-98c3-09d48ed6dec2"
Jul 13 19:45:42.891740 ip-10-0-128-215 kubenswrapper[2565]: E0713 19:45:42.891413    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mhsdz" podUID="762ad0f9-19a6-4a36-b4eb-a50f7291ab13"
Jul 13 19:45:43.500807 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:45:43.500773    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mhsdz"
Jul 13 19:45:43.501013 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:45:43.500773    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8gwt"
Jul 13 19:46:42.193147 ip-10-0-128-215 kubenswrapper[2565]: E0713 19:46:42.193088    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dmww7f" podUID="634b78c4-751c-4880-9f6c-1ea69b737987"
Jul 13 19:46:42.794262 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:46:42.794225    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dmww7f"
Jul 13 19:47:46.501981 ip-10-0-128-215 kubenswrapper[2565]: E0713 19:47:46.501934    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mhsdz" podUID="762ad0f9-19a6-4a36-b4eb-a50f7291ab13"
Jul 13 19:47:46.501981 ip-10-0-128-215 kubenswrapper[2565]: E0713 19:47:46.501931    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8gwt" podUID="74e7b5d4-9d6e-4736-98c3-09d48ed6dec2"
Jul 13 19:47:47.115028 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:47:47.114997    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8gwt"
Jul 13 19:47:47.115028 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:47:47.115011    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mhsdz"
Jul 13 19:48:21.451540 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:48:21.451497    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"]
Jul 13 19:48:21.455192 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:48:21.455174    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 13 19:48:21.458952 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:48:21.458928    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-pvc-router-manage-2577e7-b09f84b1-dockercfg-8xddh\""
Jul 13 19:48:21.459109 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:48:21.458933    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv1dc4269d1ada5f2d28562215d180c57f-kserve-self-signed-certs\""
Jul 13 19:48:21.467295 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:48:21.467268    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"]
Jul 13 19:48:21.588420 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:48:21.588379    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4b3b3ac-0484-42d4-a875-f74db7379ea1-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"a4b3b3ac-0484-42d4-a875-f74db7379ea1\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 13 19:48:21.588420 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:48:21.588424    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a4b3b3ac-0484-42d4-a875-f74db7379ea1-dshm\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"a4b3b3ac-0484-42d4-a875-f74db7379ea1\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 13 19:48:21.588656 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:48:21.588452    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a4b3b3ac-0484-42d4-a875-f74db7379ea1-tls-certs\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"a4b3b3ac-0484-42d4-a875-f74db7379ea1\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 13 19:48:21.588656 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:48:21.588554    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a4b3b3ac-0484-42d4-a875-f74db7379ea1-home\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"a4b3b3ac-0484-42d4-a875-f74db7379ea1\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 13 19:48:21.588656 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:48:21.588643    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-986v7\" (UniqueName: \"kubernetes.io/projected/a4b3b3ac-0484-42d4-a875-f74db7379ea1-kube-api-access-986v7\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"a4b3b3ac-0484-42d4-a875-f74db7379ea1\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 13 19:48:21.588802 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:48:21.588670    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a4b3b3ac-0484-42d4-a875-f74db7379ea1-model-cache\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"a4b3b3ac-0484-42d4-a875-f74db7379ea1\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 13 19:48:21.689682 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:48:21.689644    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a4b3b3ac-0484-42d4-a875-f74db7379ea1-model-cache\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"a4b3b3ac-0484-42d4-a875-f74db7379ea1\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 13 19:48:21.689901 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:48:21.689786    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4b3b3ac-0484-42d4-a875-f74db7379ea1-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"a4b3b3ac-0484-42d4-a875-f74db7379ea1\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 13 19:48:21.689901 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:48:21.689827    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a4b3b3ac-0484-42d4-a875-f74db7379ea1-dshm\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"a4b3b3ac-0484-42d4-a875-f74db7379ea1\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 13 19:48:21.689984 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:48:21.689943    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a4b3b3ac-0484-42d4-a875-f74db7379ea1-tls-certs\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"a4b3b3ac-0484-42d4-a875-f74db7379ea1\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 13 19:48:21.690026 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:48:21.690006    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a4b3b3ac-0484-42d4-a875-f74db7379ea1-home\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"a4b3b3ac-0484-42d4-a875-f74db7379ea1\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 13 19:48:21.690094 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:48:21.690069    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-986v7\" (UniqueName: \"kubernetes.io/projected/a4b3b3ac-0484-42d4-a875-f74db7379ea1-kube-api-access-986v7\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"a4b3b3ac-0484-42d4-a875-f74db7379ea1\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 13 19:48:21.690094 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:48:21.690088    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a4b3b3ac-0484-42d4-a875-f74db7379ea1-model-cache\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"a4b3b3ac-0484-42d4-a875-f74db7379ea1\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 13 19:48:21.690286 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:48:21.690125    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4b3b3ac-0484-42d4-a875-f74db7379ea1-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"a4b3b3ac-0484-42d4-a875-f74db7379ea1\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 13 19:48:21.690346 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:48:21.690285    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a4b3b3ac-0484-42d4-a875-f74db7379ea1-home\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"a4b3b3ac-0484-42d4-a875-f74db7379ea1\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 13 19:48:21.692165 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:48:21.692143    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a4b3b3ac-0484-42d4-a875-f74db7379ea1-dshm\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"a4b3b3ac-0484-42d4-a875-f74db7379ea1\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 13 19:48:21.692414 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:48:21.692396    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a4b3b3ac-0484-42d4-a875-f74db7379ea1-tls-certs\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"a4b3b3ac-0484-42d4-a875-f74db7379ea1\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 13 19:48:21.699374 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:48:21.699342    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-986v7\" (UniqueName: \"kubernetes.io/projected/a4b3b3ac-0484-42d4-a875-f74db7379ea1-kube-api-access-986v7\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"a4b3b3ac-0484-42d4-a875-f74db7379ea1\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 13 19:48:45.795599 ip-10-0-128-215 kubenswrapper[2565]: E0713 19:48:45.795500    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dmww7f" podUID="634b78c4-751c-4880-9f6c-1ea69b737987"
Jul 13 19:48:46.409749 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:48:46.409689    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dmww7f"
Jul 13 19:49:21.541862 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:49:21.541828    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Jul 13 19:49:36.566859 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:49:36.566813    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="a46fc802-362c-4cd4-9d04-dd1ae4aa348f" containerName="main" containerID="cri-o://b05922c0285cbc990c2f3d078de1c236fe99964dfb0c2e7c9fb0f011df535f30" gracePeriod=45
Jul 13 19:49:36.816563 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:49:36.816511    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="a46fc802-362c-4cd4-9d04-dd1ae4aa348f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": dial tcp 10.134.0.62:8000: connect: connection refused"
Jul 13 19:49:37.243694 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:49:37.243664    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 13 19:49:37.387812 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:49:37.387685    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a46fc802-362c-4cd4-9d04-dd1ae4aa348f-tmp-dir\") pod \"a46fc802-362c-4cd4-9d04-dd1ae4aa348f\" (UID: \"a46fc802-362c-4cd4-9d04-dd1ae4aa348f\") "
Jul 13 19:49:37.387812 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:49:37.387761    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mdq2f\" (UniqueName: \"kubernetes.io/projected/a46fc802-362c-4cd4-9d04-dd1ae4aa348f-kube-api-access-mdq2f\") pod \"a46fc802-362c-4cd4-9d04-dd1ae4aa348f\" (UID: \"a46fc802-362c-4cd4-9d04-dd1ae4aa348f\") "
Jul 13 19:49:37.387812 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:49:37.387793    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a46fc802-362c-4cd4-9d04-dd1ae4aa348f-model-cache\") pod \"a46fc802-362c-4cd4-9d04-dd1ae4aa348f\" (UID: \"a46fc802-362c-4cd4-9d04-dd1ae4aa348f\") "
Jul 13 19:49:37.388106 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:49:37.387884    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a46fc802-362c-4cd4-9d04-dd1ae4aa348f-tls-certs\") pod \"a46fc802-362c-4cd4-9d04-dd1ae4aa348f\" (UID: \"a46fc802-362c-4cd4-9d04-dd1ae4aa348f\") "
Jul 13 19:49:37.388106 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:49:37.387913    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a46fc802-362c-4cd4-9d04-dd1ae4aa348f-home\") pod \"a46fc802-362c-4cd4-9d04-dd1ae4aa348f\" (UID: \"a46fc802-362c-4cd4-9d04-dd1ae4aa348f\") "
Jul 13 19:49:37.388106 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:49:37.387990    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a46fc802-362c-4cd4-9d04-dd1ae4aa348f-kserve-provision-location\") pod \"a46fc802-362c-4cd4-9d04-dd1ae4aa348f\" (UID: \"a46fc802-362c-4cd4-9d04-dd1ae4aa348f\") "
Jul 13 19:49:37.388106 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:49:37.388039    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a46fc802-362c-4cd4-9d04-dd1ae4aa348f-dshm\") pod \"a46fc802-362c-4cd4-9d04-dd1ae4aa348f\" (UID: \"a46fc802-362c-4cd4-9d04-dd1ae4aa348f\") "
Jul 13 19:49:37.388311 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:49:37.388099    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a46fc802-362c-4cd4-9d04-dd1ae4aa348f-model-cache" (OuterVolumeSpecName: "model-cache") pod "a46fc802-362c-4cd4-9d04-dd1ae4aa348f" (UID: "a46fc802-362c-4cd4-9d04-dd1ae4aa348f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:49:37.388420 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:49:37.388402    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a46fc802-362c-4cd4-9d04-dd1ae4aa348f-model-cache\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 13 19:49:37.388886 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:49:37.388853    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a46fc802-362c-4cd4-9d04-dd1ae4aa348f-home" (OuterVolumeSpecName: "home") pod "a46fc802-362c-4cd4-9d04-dd1ae4aa348f" (UID: "a46fc802-362c-4cd4-9d04-dd1ae4aa348f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:49:37.390301 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:49:37.390270    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a46fc802-362c-4cd4-9d04-dd1ae4aa348f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a46fc802-362c-4cd4-9d04-dd1ae4aa348f" (UID: "a46fc802-362c-4cd4-9d04-dd1ae4aa348f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 19:49:37.390395 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:49:37.390335    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a46fc802-362c-4cd4-9d04-dd1ae4aa348f-kube-api-access-mdq2f" (OuterVolumeSpecName: "kube-api-access-mdq2f") pod "a46fc802-362c-4cd4-9d04-dd1ae4aa348f" (UID: "a46fc802-362c-4cd4-9d04-dd1ae4aa348f"). InnerVolumeSpecName "kube-api-access-mdq2f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 19:49:37.390591 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:49:37.390573    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a46fc802-362c-4cd4-9d04-dd1ae4aa348f-dshm" (OuterVolumeSpecName: "dshm") pod "a46fc802-362c-4cd4-9d04-dd1ae4aa348f" (UID: "a46fc802-362c-4cd4-9d04-dd1ae4aa348f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:49:37.400599 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:49:37.400575    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a46fc802-362c-4cd4-9d04-dd1ae4aa348f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a46fc802-362c-4cd4-9d04-dd1ae4aa348f" (UID: "a46fc802-362c-4cd4-9d04-dd1ae4aa348f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:49:37.457327 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:49:37.457284    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a46fc802-362c-4cd4-9d04-dd1ae4aa348f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a46fc802-362c-4cd4-9d04-dd1ae4aa348f" (UID: "a46fc802-362c-4cd4-9d04-dd1ae4aa348f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 19:49:37.489139 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:49:37.489104    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a46fc802-362c-4cd4-9d04-dd1ae4aa348f-home\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 13 19:49:37.489139 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:49:37.489138    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a46fc802-362c-4cd4-9d04-dd1ae4aa348f-kserve-provision-location\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 13 19:49:37.489139 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:49:37.489147    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a46fc802-362c-4cd4-9d04-dd1ae4aa348f-dshm\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 13 19:49:37.489355 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:49:37.489155    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a46fc802-362c-4cd4-9d04-dd1ae4aa348f-tmp-dir\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 13 19:49:37.489355 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:49:37.489165    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mdq2f\" (UniqueName: \"kubernetes.io/projected/a46fc802-362c-4cd4-9d04-dd1ae4aa348f-kube-api-access-mdq2f\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 13 19:49:37.489355 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:49:37.489173    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a46fc802-362c-4cd4-9d04-dd1ae4aa348f-tls-certs\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 13 19:49:37.675621 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:49:37.675538    2565 generic.go:356] "Generic (PLEG): container finished" podID="a46fc802-362c-4cd4-9d04-dd1ae4aa348f" containerID="b05922c0285cbc990c2f3d078de1c236fe99964dfb0c2e7c9fb0f011df535f30" exitCode=0
Jul 13 19:49:37.675621 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:49:37.675603    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 13 19:49:37.676076 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:49:37.675622    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"a46fc802-362c-4cd4-9d04-dd1ae4aa348f","Type":"ContainerDied","Data":"b05922c0285cbc990c2f3d078de1c236fe99964dfb0c2e7c9fb0f011df535f30"}
Jul 13 19:49:37.676076 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:49:37.675727    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"a46fc802-362c-4cd4-9d04-dd1ae4aa348f","Type":"ContainerDied","Data":"da107e502c6db33b45e1fe0095ed93bbca199eaaa5d5c1e49129a5be10a87d6c"}
Jul 13 19:49:37.676076 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:49:37.675751    2565 scope.go:117] "RemoveContainer" containerID="b05922c0285cbc990c2f3d078de1c236fe99964dfb0c2e7c9fb0f011df535f30"
Jul 13 19:49:37.685331 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:49:37.685313    2565 scope.go:117] "RemoveContainer" containerID="3753f29fa3e4ce934c7ec925c95652880b00ba6427ee41ff840cabe2bb6e6e1b"
Jul 13 19:49:37.693530 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:49:37.693515    2565 scope.go:117] "RemoveContainer" containerID="b05922c0285cbc990c2f3d078de1c236fe99964dfb0c2e7c9fb0f011df535f30"
Jul 13 19:49:37.693800 ip-10-0-128-215 kubenswrapper[2565]: E0713 19:49:37.693782    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b05922c0285cbc990c2f3d078de1c236fe99964dfb0c2e7c9fb0f011df535f30\": container with ID starting with b05922c0285cbc990c2f3d078de1c236fe99964dfb0c2e7c9fb0f011df535f30 not found: ID does not exist" containerID="b05922c0285cbc990c2f3d078de1c236fe99964dfb0c2e7c9fb0f011df535f30"
Jul 13 19:49:37.693873 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:49:37.693806    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b05922c0285cbc990c2f3d078de1c236fe99964dfb0c2e7c9fb0f011df535f30"} err="failed to get container status \"b05922c0285cbc990c2f3d078de1c236fe99964dfb0c2e7c9fb0f011df535f30\": rpc error: code = NotFound desc = could not find container \"b05922c0285cbc990c2f3d078de1c236fe99964dfb0c2e7c9fb0f011df535f30\": container with ID starting with b05922c0285cbc990c2f3d078de1c236fe99964dfb0c2e7c9fb0f011df535f30 not found: ID does not exist"
Jul 13 19:49:37.693873 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:49:37.693823    2565 scope.go:117] "RemoveContainer" containerID="3753f29fa3e4ce934c7ec925c95652880b00ba6427ee41ff840cabe2bb6e6e1b"
Jul 13 19:49:37.694034 ip-10-0-128-215 kubenswrapper[2565]: E0713 19:49:37.694019    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3753f29fa3e4ce934c7ec925c95652880b00ba6427ee41ff840cabe2bb6e6e1b\": container with ID starting with 3753f29fa3e4ce934c7ec925c95652880b00ba6427ee41ff840cabe2bb6e6e1b not found: ID does not exist" containerID="3753f29fa3e4ce934c7ec925c95652880b00ba6427ee41ff840cabe2bb6e6e1b"
Jul 13 19:49:37.694075 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:49:37.694039    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3753f29fa3e4ce934c7ec925c95652880b00ba6427ee41ff840cabe2bb6e6e1b"} err="failed to get container status \"3753f29fa3e4ce934c7ec925c95652880b00ba6427ee41ff840cabe2bb6e6e1b\": rpc error: code = NotFound desc = could not find container \"3753f29fa3e4ce934c7ec925c95652880b00ba6427ee41ff840cabe2bb6e6e1b\": container with ID starting with 3753f29fa3e4ce934c7ec925c95652880b00ba6427ee41ff840cabe2bb6e6e1b not found: ID does not exist"
Jul 13 19:49:37.697806 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:49:37.697786    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Jul 13 19:49:37.701166 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:49:37.701147    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Jul 13 19:49:38.048181 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:49:38.048149    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a46fc802-362c-4cd4-9d04-dd1ae4aa348f" path="/var/lib/kubelet/pods/a46fc802-362c-4cd4-9d04-dd1ae4aa348f/volumes"
Jul 13 19:49:50.116273 ip-10-0-128-215 kubenswrapper[2565]: E0713 19:49:50.116218    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8gwt" podUID="74e7b5d4-9d6e-4736-98c3-09d48ed6dec2"
Jul 13 19:49:50.116809 ip-10-0-128-215 kubenswrapper[2565]: E0713 19:49:50.116306    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mhsdz" podUID="762ad0f9-19a6-4a36-b4eb-a50f7291ab13"
Jul 13 19:49:50.738772 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:49:50.738740    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8gwt"
Jul 13 19:49:50.738983 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:49:50.738742    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mhsdz"
Jul 13 19:50:14.244564 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:50:14.244537    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2bscc_2f0655bd-4a09-4314-8c1b-e30191eaec0e/ovn-acl-logging/0.log"
Jul 13 19:50:14.251721 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:50:14.251679    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2bscc_2f0655bd-4a09-4314-8c1b-e30191eaec0e/ovn-acl-logging/0.log"
Jul 13 19:50:24.466060 ip-10-0-128-215 kubenswrapper[2565]: E0713 19:50:24.466008    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" podUID="a4b3b3ac-0484-42d4-a875-f74db7379ea1"
Jul 13 19:50:24.910334 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:50:24.910302    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 13 19:50:49.411012 ip-10-0-128-215 kubenswrapper[2565]: E0713 19:50:49.410948    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dmww7f" podUID="634b78c4-751c-4880-9f6c-1ea69b737987"
Jul 13 19:50:50.037185 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:50:50.037154    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dmww7f"
Jul 13 19:51:53.739517 ip-10-0-128-215 kubenswrapper[2565]: E0713 19:51:53.739472    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8gwt" podUID="74e7b5d4-9d6e-4736-98c3-09d48ed6dec2"
Jul 13 19:51:53.739517 ip-10-0-128-215 kubenswrapper[2565]: E0713 19:51:53.739486    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mhsdz" podUID="762ad0f9-19a6-4a36-b4eb-a50f7291ab13"
Jul 13 19:51:54.341161 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:51:54.341131    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8gwt"
Jul 13 19:51:54.341338 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:51:54.341134    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mhsdz"
Jul 13 19:52:27.911902 ip-10-0-128-215 kubenswrapper[2565]: E0713 19:52:27.911840    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" podUID="a4b3b3ac-0484-42d4-a875-f74db7379ea1"
Jul 13 19:52:28.516156 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:52:28.516123    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 13 19:52:53.038809 ip-10-0-128-215 kubenswrapper[2565]: E0713 19:52:53.038758    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dmww7f" podUID="634b78c4-751c-4880-9f6c-1ea69b737987"
Jul 13 19:52:53.636669 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:52:53.636632    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dmww7f"
Jul 13 19:53:57.342631 ip-10-0-128-215 kubenswrapper[2565]: E0713 19:53:57.342576    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mhsdz" podUID="762ad0f9-19a6-4a36-b4eb-a50f7291ab13"
Jul 13 19:53:57.342631 ip-10-0-128-215 kubenswrapper[2565]: E0713 19:53:57.342576    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8gwt" podUID="74e7b5d4-9d6e-4736-98c3-09d48ed6dec2"
Jul 13 19:53:57.953025 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:53:57.952988    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8gwt"
Jul 13 19:53:57.953247 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:53:57.952996    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mhsdz"
Jul 13 19:54:31.517696 ip-10-0-128-215 kubenswrapper[2565]: E0713 19:54:31.517642    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" podUID="a4b3b3ac-0484-42d4-a875-f74db7379ea1"
Jul 13 19:54:32.114349 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:54:32.114315    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 13 19:54:56.638052 ip-10-0-128-215 kubenswrapper[2565]: E0713 19:54:56.637997    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dmww7f" podUID="634b78c4-751c-4880-9f6c-1ea69b737987"
Jul 13 19:54:57.235944 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:54:57.235908    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dmww7f"
Jul 13 19:55:14.281433 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:55:14.281395    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2bscc_2f0655bd-4a09-4314-8c1b-e30191eaec0e/ovn-acl-logging/0.log"
Jul 13 19:55:14.284059 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:55:14.284033    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2bscc_2f0655bd-4a09-4314-8c1b-e30191eaec0e/ovn-acl-logging/0.log"
Jul 13 19:56:00.954602 ip-10-0-128-215 kubenswrapper[2565]: E0713 19:56:00.954555    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8gwt" podUID="74e7b5d4-9d6e-4736-98c3-09d48ed6dec2"
Jul 13 19:56:00.954602 ip-10-0-128-215 kubenswrapper[2565]: E0713 19:56:00.954567    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mhsdz" podUID="762ad0f9-19a6-4a36-b4eb-a50f7291ab13"
Jul 13 19:56:01.545137 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:56:01.545108    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8gwt"
Jul 13 19:56:01.545328 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:56:01.545108    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mhsdz"
Jul 13 19:56:35.115465 ip-10-0-128-215 kubenswrapper[2565]: E0713 19:56:35.115420    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" podUID="a4b3b3ac-0484-42d4-a875-f74db7379ea1"
Jul 13 19:56:35.723979 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:56:35.723945    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 13 19:57:00.237104 ip-10-0-128-215 kubenswrapper[2565]: E0713 19:57:00.237053    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dmww7f" podUID="634b78c4-751c-4880-9f6c-1ea69b737987"
Jul 13 19:57:00.847004 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:57:00.846968    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dmww7f"
Jul 13 19:58:04.546458 ip-10-0-128-215 kubenswrapper[2565]: E0713 19:58:04.546397    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mhsdz" podUID="762ad0f9-19a6-4a36-b4eb-a50f7291ab13"
Jul 13 19:58:04.546458 ip-10-0-128-215 kubenswrapper[2565]: E0713 19:58:04.546404    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8gwt" podUID="74e7b5d4-9d6e-4736-98c3-09d48ed6dec2"
Jul 13 19:58:05.158483 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:58:05.158453    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8gwt"
Jul 13 19:58:05.158674 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:58:05.158457    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mhsdz"
Jul 13 19:58:38.725005 ip-10-0-128-215 kubenswrapper[2565]: E0713 19:58:38.724960    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" podUID="a4b3b3ac-0484-42d4-a875-f74db7379ea1"
Jul 13 19:58:39.331006 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:58:39.330976    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 13 19:59:03.848547 ip-10-0-128-215 kubenswrapper[2565]: E0713 19:59:03.848431    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dmww7f" podUID="634b78c4-751c-4880-9f6c-1ea69b737987"
Jul 13 19:59:04.464249 ip-10-0-128-215 kubenswrapper[2565]: I0713 19:59:04.464214    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dmww7f"
Jul 13 20:00:08.159863 ip-10-0-128-215 kubenswrapper[2565]: E0713 20:00:08.159799    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8gwt" podUID="74e7b5d4-9d6e-4736-98c3-09d48ed6dec2"
Jul 13 20:00:08.159863 ip-10-0-128-215 kubenswrapper[2565]: E0713 20:00:08.159799    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mhsdz" podUID="762ad0f9-19a6-4a36-b4eb-a50f7291ab13"
Jul 13 20:00:08.774113 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:00:08.774086    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8gwt"
Jul 13 20:00:08.774309 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:00:08.774094    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mhsdz"
Jul 13 20:00:14.311671 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:00:14.311640    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2bscc_2f0655bd-4a09-4314-8c1b-e30191eaec0e/ovn-acl-logging/0.log"
Jul 13 20:00:14.316805 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:00:14.316781    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2bscc_2f0655bd-4a09-4314-8c1b-e30191eaec0e/ovn-acl-logging/0.log"
Jul 13 20:00:42.331981 ip-10-0-128-215 kubenswrapper[2565]: E0713 20:00:42.331928    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" podUID="a4b3b3ac-0484-42d4-a875-f74db7379ea1"
Jul 13 20:00:42.941404 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:00:42.941364    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 13 20:01:07.465896 ip-10-0-128-215 kubenswrapper[2565]: E0713 20:01:07.465845    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dmww7f" podUID="634b78c4-751c-4880-9f6c-1ea69b737987"
Jul 13 20:01:08.066627 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:01:08.066585    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dmww7f"
Jul 13 20:02:11.775068 ip-10-0-128-215 kubenswrapper[2565]: E0713 20:02:11.774968    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mhsdz" podUID="762ad0f9-19a6-4a36-b4eb-a50f7291ab13"
Jul 13 20:02:11.775068 ip-10-0-128-215 kubenswrapper[2565]: E0713 20:02:11.774991    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8gwt" podUID="74e7b5d4-9d6e-4736-98c3-09d48ed6dec2"
Jul 13 20:02:12.369545 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:02:12.369511    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mhsdz"
Jul 13 20:02:12.369757 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:02:12.369512    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8gwt"
Jul 13 20:02:45.942505 ip-10-0-128-215 kubenswrapper[2565]: E0713 20:02:45.942447    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" podUID="a4b3b3ac-0484-42d4-a875-f74db7379ea1"
Jul 13 20:02:46.536187 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:02:46.536155    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 13 20:03:11.067725 ip-10-0-128-215 kubenswrapper[2565]: E0713 20:03:11.067648    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dmww7f" podUID="634b78c4-751c-4880-9f6c-1ea69b737987"
Jul 13 20:03:11.658593 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:03:11.658561    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dmww7f"
Jul 13 20:04:15.370249 ip-10-0-128-215 kubenswrapper[2565]: E0713 20:04:15.370195    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8gwt" podUID="74e7b5d4-9d6e-4736-98c3-09d48ed6dec2"
Jul 13 20:04:15.370786 ip-10-0-128-215 kubenswrapper[2565]: E0713 20:04:15.370259    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mhsdz" podUID="762ad0f9-19a6-4a36-b4eb-a50f7291ab13"
Jul 13 20:04:15.976270 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:04:15.976233    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8gwt"
Jul 13 20:04:15.976472 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:04:15.976233    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mhsdz"
Jul 13 20:04:49.537246 ip-10-0-128-215 kubenswrapper[2565]: E0713 20:04:49.537196    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" podUID="a4b3b3ac-0484-42d4-a875-f74db7379ea1"
Jul 13 20:04:50.150078 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:04:50.150044    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 13 20:05:14.343202 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:05:14.343118    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2bscc_2f0655bd-4a09-4314-8c1b-e30191eaec0e/ovn-acl-logging/0.log"
Jul 13 20:05:14.349228 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:05:14.349205    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2bscc_2f0655bd-4a09-4314-8c1b-e30191eaec0e/ovn-acl-logging/0.log"
Jul 13 20:05:14.659765 ip-10-0-128-215 kubenswrapper[2565]: E0713 20:05:14.659642    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dmww7f" podUID="634b78c4-751c-4880-9f6c-1ea69b737987"
Jul 13 20:05:15.269779 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:05:15.269747    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dmww7f"
Jul 13 20:06:18.977796 ip-10-0-128-215 kubenswrapper[2565]: E0713 20:06:18.977747    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8gwt" podUID="74e7b5d4-9d6e-4736-98c3-09d48ed6dec2"
Jul 13 20:06:18.977796 ip-10-0-128-215 kubenswrapper[2565]: E0713 20:06:18.977764    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mhsdz" podUID="762ad0f9-19a6-4a36-b4eb-a50f7291ab13"
Jul 13 20:06:19.580228 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:06:19.580193    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mhsdz"
Jul 13 20:06:19.580450 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:06:19.580193    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8gwt"
Jul 13 20:06:53.151695 ip-10-0-128-215 kubenswrapper[2565]: E0713 20:06:53.151599    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" podUID="a4b3b3ac-0484-42d4-a875-f74db7379ea1"
Jul 13 20:06:53.752926 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:06:53.752894    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 13 20:07:18.271187 ip-10-0-128-215 kubenswrapper[2565]: E0713 20:07:18.271123    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dmww7f" podUID="634b78c4-751c-4880-9f6c-1ea69b737987"
Jul 13 20:07:18.876080 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:07:18.876046    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dmww7f"
Jul 13 20:08:22.581417 ip-10-0-128-215 kubenswrapper[2565]: E0713 20:08:22.581354    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mhsdz" podUID="762ad0f9-19a6-4a36-b4eb-a50f7291ab13"
Jul 13 20:08:22.581417 ip-10-0-128-215 kubenswrapper[2565]: E0713 20:08:22.581354    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8gwt" podUID="74e7b5d4-9d6e-4736-98c3-09d48ed6dec2"
Jul 13 20:08:23.193950 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:08:23.193914    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8gwt"
Jul 13 20:08:23.194140 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:08:23.193914    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mhsdz"
Jul 13 20:08:56.754376 ip-10-0-128-215 kubenswrapper[2565]: E0713 20:08:56.754337    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" podUID="a4b3b3ac-0484-42d4-a875-f74db7379ea1"
Jul 13 20:08:57.359115 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:08:57.359078    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 13 20:09:21.877040 ip-10-0-128-215 kubenswrapper[2565]: E0713 20:09:21.877000    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dmww7f" podUID="634b78c4-751c-4880-9f6c-1ea69b737987"
Jul 13 20:09:22.481020 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:09:22.480989    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dmww7f"
Jul 13 20:10:14.374607 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:10:14.374573    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2bscc_2f0655bd-4a09-4314-8c1b-e30191eaec0e/ovn-acl-logging/0.log"
Jul 13 20:10:14.383800 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:10:14.383773    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2bscc_2f0655bd-4a09-4314-8c1b-e30191eaec0e/ovn-acl-logging/0.log"
Jul 13 20:10:26.195165 ip-10-0-128-215 kubenswrapper[2565]: E0713 20:10:26.195099    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8gwt" podUID="74e7b5d4-9d6e-4736-98c3-09d48ed6dec2"
Jul 13 20:10:26.195165 ip-10-0-128-215 kubenswrapper[2565]: E0713 20:10:26.195099    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mhsdz" podUID="762ad0f9-19a6-4a36-b4eb-a50f7291ab13"
Jul 13 20:10:26.792285 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:10:26.792252    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8gwt"
Jul 13 20:10:26.792285 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:10:26.792284    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mhsdz"
Jul 13 20:11:00.359691 ip-10-0-128-215 kubenswrapper[2565]: E0713 20:11:00.359653    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" podUID="a4b3b3ac-0484-42d4-a875-f74db7379ea1"
Jul 13 20:11:00.969539 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:11:00.969510    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 13 20:11:25.482071 ip-10-0-128-215 kubenswrapper[2565]: E0713 20:11:25.482012    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dmww7f" podUID="634b78c4-751c-4880-9f6c-1ea69b737987"
Jul 13 20:11:26.086025 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:11:26.085991    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dmww7f"
Jul 13 20:12:29.793621 ip-10-0-128-215 kubenswrapper[2565]: E0713 20:12:29.793579    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8gwt" podUID="74e7b5d4-9d6e-4736-98c3-09d48ed6dec2"
Jul 13 20:12:29.793621 ip-10-0-128-215 kubenswrapper[2565]: E0713 20:12:29.793580    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mhsdz" podUID="762ad0f9-19a6-4a36-b4eb-a50f7291ab13"
Jul 13 20:12:30.402001 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:12:30.401972    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mhsdz"
Jul 13 20:12:30.402175 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:12:30.401974    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8gwt"
Jul 13 20:12:44.424927 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:12:44.424836    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-65cf7fd568-dnqbs"]
Jul 13 20:12:44.425393 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:12:44.425375    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a46fc802-362c-4cd4-9d04-dd1ae4aa348f" containerName="storage-initializer"
Jul 13 20:12:44.425447 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:12:44.425396    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="a46fc802-362c-4cd4-9d04-dd1ae4aa348f" containerName="storage-initializer"
Jul 13 20:12:44.425447 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:12:44.425406    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a46fc802-362c-4cd4-9d04-dd1ae4aa348f" containerName="main"
Jul 13 20:12:44.425447 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:12:44.425413    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="a46fc802-362c-4cd4-9d04-dd1ae4aa348f" containerName="main"
Jul 13 20:12:44.425541 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:12:44.425496    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="a46fc802-362c-4cd4-9d04-dd1ae4aa348f" containerName="main"
Jul 13 20:12:44.428730 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:12:44.428714    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-65cf7fd568-dnqbs"
Jul 13 20:12:44.431535 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:12:44.431520    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 13 20:12:44.431624 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:12:44.431519    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"prestop-hook-test-kserve-self-signed-certs\""
Jul 13 20:12:44.443388 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:12:44.443366    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-65cf7fd568-dnqbs"]
Jul 13 20:12:44.496914 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:12:44.496880    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2c2494c1-9709-4ae8-9730-67eafc2d1425-kserve-provision-location\") pod \"prestop-hook-test-kserve-65cf7fd568-dnqbs\" (UID: \"2c2494c1-9709-4ae8-9730-67eafc2d1425\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-65cf7fd568-dnqbs"
Jul 13 20:12:44.497080 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:12:44.496949    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4vb7\" (UniqueName: \"kubernetes.io/projected/2c2494c1-9709-4ae8-9730-67eafc2d1425-kube-api-access-b4vb7\") pod \"prestop-hook-test-kserve-65cf7fd568-dnqbs\" (UID: \"2c2494c1-9709-4ae8-9730-67eafc2d1425\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-65cf7fd568-dnqbs"
Jul 13 20:12:44.497080 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:12:44.496983    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c2494c1-9709-4ae8-9730-67eafc2d1425-tmp-dir\") pod \"prestop-hook-test-kserve-65cf7fd568-dnqbs\" (UID: \"2c2494c1-9709-4ae8-9730-67eafc2d1425\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-65cf7fd568-dnqbs"
Jul 13 20:12:44.497080 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:12:44.497002    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c2494c1-9709-4ae8-9730-67eafc2d1425-home\") pod \"prestop-hook-test-kserve-65cf7fd568-dnqbs\" (UID: \"2c2494c1-9709-4ae8-9730-67eafc2d1425\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-65cf7fd568-dnqbs"
Jul 13 20:12:44.497080 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:12:44.497038    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c2494c1-9709-4ae8-9730-67eafc2d1425-tls-certs\") pod \"prestop-hook-test-kserve-65cf7fd568-dnqbs\" (UID: \"2c2494c1-9709-4ae8-9730-67eafc2d1425\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-65cf7fd568-dnqbs"
Jul 13 20:12:44.497080 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:12:44.497072    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c2494c1-9709-4ae8-9730-67eafc2d1425-model-cache\") pod \"prestop-hook-test-kserve-65cf7fd568-dnqbs\" (UID: \"2c2494c1-9709-4ae8-9730-67eafc2d1425\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-65cf7fd568-dnqbs"
Jul 13 20:12:44.497310 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:12:44.497149    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c2494c1-9709-4ae8-9730-67eafc2d1425-dshm\") pod \"prestop-hook-test-kserve-65cf7fd568-dnqbs\" (UID: \"2c2494c1-9709-4ae8-9730-67eafc2d1425\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-65cf7fd568-dnqbs"
Jul 13 20:12:44.597775 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:12:44.597728    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c2494c1-9709-4ae8-9730-67eafc2d1425-model-cache\") pod \"prestop-hook-test-kserve-65cf7fd568-dnqbs\" (UID: \"2c2494c1-9709-4ae8-9730-67eafc2d1425\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-65cf7fd568-dnqbs"
Jul 13 20:12:44.597961 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:12:44.597806    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c2494c1-9709-4ae8-9730-67eafc2d1425-dshm\") pod \"prestop-hook-test-kserve-65cf7fd568-dnqbs\" (UID: \"2c2494c1-9709-4ae8-9730-67eafc2d1425\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-65cf7fd568-dnqbs"
Jul 13 20:12:44.597961 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:12:44.597832    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2c2494c1-9709-4ae8-9730-67eafc2d1425-kserve-provision-location\") pod \"prestop-hook-test-kserve-65cf7fd568-dnqbs\" (UID: \"2c2494c1-9709-4ae8-9730-67eafc2d1425\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-65cf7fd568-dnqbs"
Jul 13 20:12:44.597961 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:12:44.597897    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4vb7\" (UniqueName: \"kubernetes.io/projected/2c2494c1-9709-4ae8-9730-67eafc2d1425-kube-api-access-b4vb7\") pod \"prestop-hook-test-kserve-65cf7fd568-dnqbs\" (UID: \"2c2494c1-9709-4ae8-9730-67eafc2d1425\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-65cf7fd568-dnqbs"
Jul 13 20:12:44.597961 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:12:44.597923    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c2494c1-9709-4ae8-9730-67eafc2d1425-tmp-dir\") pod \"prestop-hook-test-kserve-65cf7fd568-dnqbs\" (UID: \"2c2494c1-9709-4ae8-9730-67eafc2d1425\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-65cf7fd568-dnqbs"
Jul 13 20:12:44.597961 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:12:44.597944    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c2494c1-9709-4ae8-9730-67eafc2d1425-home\") pod \"prestop-hook-test-kserve-65cf7fd568-dnqbs\" (UID: \"2c2494c1-9709-4ae8-9730-67eafc2d1425\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-65cf7fd568-dnqbs"
Jul 13 20:12:44.598231 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:12:44.597972    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c2494c1-9709-4ae8-9730-67eafc2d1425-tls-certs\") pod \"prestop-hook-test-kserve-65cf7fd568-dnqbs\" (UID: \"2c2494c1-9709-4ae8-9730-67eafc2d1425\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-65cf7fd568-dnqbs"
Jul 13 20:12:44.598231 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:12:44.598136    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c2494c1-9709-4ae8-9730-67eafc2d1425-model-cache\") pod \"prestop-hook-test-kserve-65cf7fd568-dnqbs\" (UID: \"2c2494c1-9709-4ae8-9730-67eafc2d1425\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-65cf7fd568-dnqbs"
Jul 13 20:12:44.598339 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:12:44.598243    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2c2494c1-9709-4ae8-9730-67eafc2d1425-kserve-provision-location\") pod \"prestop-hook-test-kserve-65cf7fd568-dnqbs\" (UID: \"2c2494c1-9709-4ae8-9730-67eafc2d1425\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-65cf7fd568-dnqbs"
Jul 13 20:12:44.598339 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:12:44.598310    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c2494c1-9709-4ae8-9730-67eafc2d1425-home\") pod \"prestop-hook-test-kserve-65cf7fd568-dnqbs\" (UID: \"2c2494c1-9709-4ae8-9730-67eafc2d1425\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-65cf7fd568-dnqbs"
Jul 13 20:12:44.598339 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:12:44.598322    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c2494c1-9709-4ae8-9730-67eafc2d1425-tmp-dir\") pod \"prestop-hook-test-kserve-65cf7fd568-dnqbs\" (UID: \"2c2494c1-9709-4ae8-9730-67eafc2d1425\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-65cf7fd568-dnqbs"
Jul 13 20:12:44.600137 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:12:44.600120    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c2494c1-9709-4ae8-9730-67eafc2d1425-dshm\") pod \"prestop-hook-test-kserve-65cf7fd568-dnqbs\" (UID: \"2c2494c1-9709-4ae8-9730-67eafc2d1425\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-65cf7fd568-dnqbs"
Jul 13 20:12:44.600483 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:12:44.600467    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c2494c1-9709-4ae8-9730-67eafc2d1425-tls-certs\") pod \"prestop-hook-test-kserve-65cf7fd568-dnqbs\" (UID: \"2c2494c1-9709-4ae8-9730-67eafc2d1425\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-65cf7fd568-dnqbs"
Jul 13 20:12:44.607061 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:12:44.607044    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4vb7\" (UniqueName: \"kubernetes.io/projected/2c2494c1-9709-4ae8-9730-67eafc2d1425-kube-api-access-b4vb7\") pod \"prestop-hook-test-kserve-65cf7fd568-dnqbs\" (UID: \"2c2494c1-9709-4ae8-9730-67eafc2d1425\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-65cf7fd568-dnqbs"
Jul 13 20:12:44.745190 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:12:44.745103    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-gv2sm\""
Jul 13 20:12:44.753200 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:12:44.753180    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-65cf7fd568-dnqbs"
Jul 13 20:12:44.896333 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:12:44.896298    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-65cf7fd568-dnqbs"]
Jul 13 20:12:44.897750 ip-10-0-128-215 kubenswrapper[2565]: W0713 20:12:44.897718    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c2494c1_9709_4ae8_9730_67eafc2d1425.slice/crio-cdc1da4ea5174b0f0e05bc9d14ab263cebd9dbed030448538e109ea11f08c3ec WatchSource:0}: Error finding container cdc1da4ea5174b0f0e05bc9d14ab263cebd9dbed030448538e109ea11f08c3ec: Status 404 returned error can't find the container with id cdc1da4ea5174b0f0e05bc9d14ab263cebd9dbed030448538e109ea11f08c3ec
Jul 13 20:12:45.474778 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:12:45.474740    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-65cf7fd568-dnqbs" event={"ID":"2c2494c1-9709-4ae8-9730-67eafc2d1425","Type":"ContainerStarted","Data":"7c2101ac4dcf246c0eadc633a0620689a545743613b3cda23726ff0a26aac6c8"}
Jul 13 20:12:45.475199 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:12:45.474798    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-65cf7fd568-dnqbs" event={"ID":"2c2494c1-9709-4ae8-9730-67eafc2d1425","Type":"ContainerStarted","Data":"cdc1da4ea5174b0f0e05bc9d14ab263cebd9dbed030448538e109ea11f08c3ec"}
Jul 13 20:12:49.503900 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:12:49.503861    2565 generic.go:356] "Generic (PLEG): container finished" podID="2c2494c1-9709-4ae8-9730-67eafc2d1425" containerID="7c2101ac4dcf246c0eadc633a0620689a545743613b3cda23726ff0a26aac6c8" exitCode=0
Jul 13 20:12:49.504360 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:12:49.503906    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-65cf7fd568-dnqbs" event={"ID":"2c2494c1-9709-4ae8-9730-67eafc2d1425","Type":"ContainerDied","Data":"7c2101ac4dcf246c0eadc633a0620689a545743613b3cda23726ff0a26aac6c8"}
Jul 13 20:12:50.512253 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:12:50.512218    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-65cf7fd568-dnqbs" event={"ID":"2c2494c1-9709-4ae8-9730-67eafc2d1425","Type":"ContainerStarted","Data":"4d62496ec9c79628bfe281a538282d09511a807c74a6bf358784d630dd0c2202"}
Jul 13 20:12:50.542138 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:12:50.542082    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-65cf7fd568-dnqbs" podStartSLOduration=2.636192551 podStartE2EDuration="6.542066325s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.905873774s" isStatefulPod=false podCreationTimestamp="2026-07-13 20:12:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 20:12:50.538126837 +0000 UTC m=+4957.063260263" watchObservedRunningTime="2026-07-13 20:12:50.542066325 +0000 UTC m=+4957.067199739"
Jul 13 20:12:50.753583 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:12:50.753547    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-65cf7fd568-dnqbs"
Jul 13 20:12:54.753942 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:12:54.753899    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-65cf7fd568-dnqbs"
Jul 13 20:12:54.755972 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:12:54.755937    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-65cf7fd568-dnqbs" podUID="2c2494c1-9709-4ae8-9730-67eafc2d1425" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/health\": dial tcp 10.134.0.64:8000: connect: connection refused"
Jul 13 20:13:03.970831 ip-10-0-128-215 kubenswrapper[2565]: E0713 20:13:03.970767    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" podUID="a4b3b3ac-0484-42d4-a875-f74db7379ea1"
Jul 13 20:13:04.585789 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:13:04.585755    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 13 20:13:04.754237 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:13:04.754187    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-65cf7fd568-dnqbs" podUID="2c2494c1-9709-4ae8-9730-67eafc2d1425" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/health\": dial tcp 10.134.0.64:8000: connect: connection refused"
Jul 13 20:13:14.753801 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:13:14.753751    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-65cf7fd568-dnqbs" podUID="2c2494c1-9709-4ae8-9730-67eafc2d1425" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/health\": dial tcp 10.134.0.64:8000: connect: connection refused"
Jul 13 20:13:24.754774 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:13:24.754675    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-65cf7fd568-dnqbs" podUID="2c2494c1-9709-4ae8-9730-67eafc2d1425" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/health\": dial tcp 10.134.0.64:8000: connect: connection refused"
Jul 13 20:13:29.086728 ip-10-0-128-215 kubenswrapper[2565]: E0713 20:13:29.086669    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dmww7f" podUID="634b78c4-751c-4880-9f6c-1ea69b737987"
Jul 13 20:13:29.717261 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:13:29.717227    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dmww7f"
Jul 13 20:13:34.753785 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:13:34.753738    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-65cf7fd568-dnqbs" podUID="2c2494c1-9709-4ae8-9730-67eafc2d1425" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/health\": dial tcp 10.134.0.64:8000: connect: connection refused"
Jul 13 20:13:44.754550 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:13:44.754507    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-65cf7fd568-dnqbs" podUID="2c2494c1-9709-4ae8-9730-67eafc2d1425" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/health\": dial tcp 10.134.0.64:8000: connect: connection refused"
Jul 13 20:13:54.754089 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:13:54.754036    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-65cf7fd568-dnqbs" podUID="2c2494c1-9709-4ae8-9730-67eafc2d1425" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/health\": dial tcp 10.134.0.64:8000: connect: connection refused"
Jul 13 20:13:59.261843 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:13:59.261794    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-65cf7fd568-dnqbs"]
Jul 13 20:14:14.298451 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:14:14.298346    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-65cf7fd568-dnqbs" podUID="2c2494c1-9709-4ae8-9730-67eafc2d1425" containerName="main" containerID="cri-o://4d62496ec9c79628bfe281a538282d09511a807c74a6bf358784d630dd0c2202" gracePeriod=45
Jul 13 20:14:33.403312 ip-10-0-128-215 kubenswrapper[2565]: E0713 20:14:33.403267    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mhsdz" podUID="762ad0f9-19a6-4a36-b4eb-a50f7291ab13"
Jul 13 20:14:33.403312 ip-10-0-128-215 kubenswrapper[2565]: E0713 20:14:33.403267    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8gwt" podUID="74e7b5d4-9d6e-4736-98c3-09d48ed6dec2"
Jul 13 20:14:34.060274 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:14:34.060238    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8gwt"
Jul 13 20:14:34.060274 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:14:34.060256    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mhsdz"
Jul 13 20:14:59.945406 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:14:59.945381    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_prestop-hook-test-kserve-65cf7fd568-dnqbs_2c2494c1-9709-4ae8-9730-67eafc2d1425/main/0.log"
Jul 13 20:14:59.947838 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:14:59.947822    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-65cf7fd568-dnqbs"
Jul 13 20:14:59.989325 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:14:59.989298    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2c2494c1-9709-4ae8-9730-67eafc2d1425-kserve-provision-location\") pod \"2c2494c1-9709-4ae8-9730-67eafc2d1425\" (UID: \"2c2494c1-9709-4ae8-9730-67eafc2d1425\") "
Jul 13 20:14:59.989479 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:14:59.989391    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c2494c1-9709-4ae8-9730-67eafc2d1425-tls-certs\") pod \"2c2494c1-9709-4ae8-9730-67eafc2d1425\" (UID: \"2c2494c1-9709-4ae8-9730-67eafc2d1425\") "
Jul 13 20:14:59.989479 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:14:59.989412    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c2494c1-9709-4ae8-9730-67eafc2d1425-home\") pod \"2c2494c1-9709-4ae8-9730-67eafc2d1425\" (UID: \"2c2494c1-9709-4ae8-9730-67eafc2d1425\") "
Jul 13 20:14:59.989479 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:14:59.989435    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c2494c1-9709-4ae8-9730-67eafc2d1425-dshm\") pod \"2c2494c1-9709-4ae8-9730-67eafc2d1425\" (UID: \"2c2494c1-9709-4ae8-9730-67eafc2d1425\") "
Jul 13 20:14:59.989658 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:14:59.989506    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c2494c1-9709-4ae8-9730-67eafc2d1425-model-cache\") pod \"2c2494c1-9709-4ae8-9730-67eafc2d1425\" (UID: \"2c2494c1-9709-4ae8-9730-67eafc2d1425\") "
Jul 13 20:14:59.989658 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:14:59.989545    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b4vb7\" (UniqueName: \"kubernetes.io/projected/2c2494c1-9709-4ae8-9730-67eafc2d1425-kube-api-access-b4vb7\") pod \"2c2494c1-9709-4ae8-9730-67eafc2d1425\" (UID: \"2c2494c1-9709-4ae8-9730-67eafc2d1425\") "
Jul 13 20:14:59.989658 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:14:59.989570    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c2494c1-9709-4ae8-9730-67eafc2d1425-tmp-dir\") pod \"2c2494c1-9709-4ae8-9730-67eafc2d1425\" (UID: \"2c2494c1-9709-4ae8-9730-67eafc2d1425\") "
Jul 13 20:14:59.990006 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:14:59.989965    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c2494c1-9709-4ae8-9730-67eafc2d1425-model-cache" (OuterVolumeSpecName: "model-cache") pod "2c2494c1-9709-4ae8-9730-67eafc2d1425" (UID: "2c2494c1-9709-4ae8-9730-67eafc2d1425"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 20:14:59.990730 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:14:59.990675    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c2494c1-9709-4ae8-9730-67eafc2d1425-home" (OuterVolumeSpecName: "home") pod "2c2494c1-9709-4ae8-9730-67eafc2d1425" (UID: "2c2494c1-9709-4ae8-9730-67eafc2d1425"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 20:14:59.991822 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:14:59.991795    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c2494c1-9709-4ae8-9730-67eafc2d1425-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2c2494c1-9709-4ae8-9730-67eafc2d1425" (UID: "2c2494c1-9709-4ae8-9730-67eafc2d1425"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 13 20:14:59.992295 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:14:59.992275    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c2494c1-9709-4ae8-9730-67eafc2d1425-kube-api-access-b4vb7" (OuterVolumeSpecName: "kube-api-access-b4vb7") pod "2c2494c1-9709-4ae8-9730-67eafc2d1425" (UID: "2c2494c1-9709-4ae8-9730-67eafc2d1425"). InnerVolumeSpecName "kube-api-access-b4vb7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 13 20:14:59.992378 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:14:59.992295    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c2494c1-9709-4ae8-9730-67eafc2d1425-dshm" (OuterVolumeSpecName: "dshm") pod "2c2494c1-9709-4ae8-9730-67eafc2d1425" (UID: "2c2494c1-9709-4ae8-9730-67eafc2d1425"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 20:15:00.008481 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:15:00.008440    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c2494c1-9709-4ae8-9730-67eafc2d1425-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2c2494c1-9709-4ae8-9730-67eafc2d1425" (UID: "2c2494c1-9709-4ae8-9730-67eafc2d1425"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 20:15:00.059263 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:15:00.059227    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c2494c1-9709-4ae8-9730-67eafc2d1425-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2c2494c1-9709-4ae8-9730-67eafc2d1425" (UID: "2c2494c1-9709-4ae8-9730-67eafc2d1425"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 13 20:15:00.090434 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:15:00.090365    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c2494c1-9709-4ae8-9730-67eafc2d1425-model-cache\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 13 20:15:00.090434 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:15:00.090391    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b4vb7\" (UniqueName: \"kubernetes.io/projected/2c2494c1-9709-4ae8-9730-67eafc2d1425-kube-api-access-b4vb7\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 13 20:15:00.090434 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:15:00.090401    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c2494c1-9709-4ae8-9730-67eafc2d1425-tmp-dir\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 13 20:15:00.090434 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:15:00.090411    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2c2494c1-9709-4ae8-9730-67eafc2d1425-kserve-provision-location\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 13 20:15:00.090434 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:15:00.090420    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c2494c1-9709-4ae8-9730-67eafc2d1425-tls-certs\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 13 20:15:00.090434 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:15:00.090427    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c2494c1-9709-4ae8-9730-67eafc2d1425-home\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 13 20:15:00.090434 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:15:00.090434    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c2494c1-9709-4ae8-9730-67eafc2d1425-dshm\") on node \"ip-10-0-128-215.ec2.internal\" DevicePath \"\""
Jul 13 20:15:00.185878 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:15:00.185853    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_prestop-hook-test-kserve-65cf7fd568-dnqbs_2c2494c1-9709-4ae8-9730-67eafc2d1425/main/0.log"
Jul 13 20:15:00.186218 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:15:00.186193    2565 generic.go:356] "Generic (PLEG): container finished" podID="2c2494c1-9709-4ae8-9730-67eafc2d1425" containerID="4d62496ec9c79628bfe281a538282d09511a807c74a6bf358784d630dd0c2202" exitCode=137
Jul 13 20:15:00.186273 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:15:00.186248    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-65cf7fd568-dnqbs"
Jul 13 20:15:00.186309 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:15:00.186266    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-65cf7fd568-dnqbs" event={"ID":"2c2494c1-9709-4ae8-9730-67eafc2d1425","Type":"ContainerDied","Data":"4d62496ec9c79628bfe281a538282d09511a807c74a6bf358784d630dd0c2202"}
Jul 13 20:15:00.186343 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:15:00.186316    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-65cf7fd568-dnqbs" event={"ID":"2c2494c1-9709-4ae8-9730-67eafc2d1425","Type":"ContainerDied","Data":"cdc1da4ea5174b0f0e05bc9d14ab263cebd9dbed030448538e109ea11f08c3ec"}
Jul 13 20:15:00.186343 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:15:00.186338    2565 scope.go:117] "RemoveContainer" containerID="4d62496ec9c79628bfe281a538282d09511a807c74a6bf358784d630dd0c2202"
Jul 13 20:15:00.195963 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:15:00.195944    2565 scope.go:117] "RemoveContainer" containerID="7c2101ac4dcf246c0eadc633a0620689a545743613b3cda23726ff0a26aac6c8"
Jul 13 20:15:00.204571 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:15:00.204553    2565 scope.go:117] "RemoveContainer" containerID="4d62496ec9c79628bfe281a538282d09511a807c74a6bf358784d630dd0c2202"
Jul 13 20:15:00.204846 ip-10-0-128-215 kubenswrapper[2565]: E0713 20:15:00.204827    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d62496ec9c79628bfe281a538282d09511a807c74a6bf358784d630dd0c2202\": container with ID starting with 4d62496ec9c79628bfe281a538282d09511a807c74a6bf358784d630dd0c2202 not found: ID does not exist" containerID="4d62496ec9c79628bfe281a538282d09511a807c74a6bf358784d630dd0c2202"
Jul 13 20:15:00.204899 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:15:00.204855    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d62496ec9c79628bfe281a538282d09511a807c74a6bf358784d630dd0c2202"} err="failed to get container status \"4d62496ec9c79628bfe281a538282d09511a807c74a6bf358784d630dd0c2202\": rpc error: code = NotFound desc = could not find container \"4d62496ec9c79628bfe281a538282d09511a807c74a6bf358784d630dd0c2202\": container with ID starting with 4d62496ec9c79628bfe281a538282d09511a807c74a6bf358784d630dd0c2202 not found: ID does not exist"
Jul 13 20:15:00.204899 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:15:00.204873    2565 scope.go:117] "RemoveContainer" containerID="7c2101ac4dcf246c0eadc633a0620689a545743613b3cda23726ff0a26aac6c8"
Jul 13 20:15:00.205099 ip-10-0-128-215 kubenswrapper[2565]: E0713 20:15:00.205081    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c2101ac4dcf246c0eadc633a0620689a545743613b3cda23726ff0a26aac6c8\": container with ID starting with 7c2101ac4dcf246c0eadc633a0620689a545743613b3cda23726ff0a26aac6c8 not found: ID does not exist" containerID="7c2101ac4dcf246c0eadc633a0620689a545743613b3cda23726ff0a26aac6c8"
Jul 13 20:15:00.205145 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:15:00.205105    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c2101ac4dcf246c0eadc633a0620689a545743613b3cda23726ff0a26aac6c8"} err="failed to get container status \"7c2101ac4dcf246c0eadc633a0620689a545743613b3cda23726ff0a26aac6c8\": rpc error: code = NotFound desc = could not find container \"7c2101ac4dcf246c0eadc633a0620689a545743613b3cda23726ff0a26aac6c8\": container with ID starting with 7c2101ac4dcf246c0eadc633a0620689a545743613b3cda23726ff0a26aac6c8 not found: ID does not exist"
Jul 13 20:15:00.209562 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:15:00.209541    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-65cf7fd568-dnqbs"]
Jul 13 20:15:00.215424 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:15:00.215404    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-65cf7fd568-dnqbs"]
Jul 13 20:15:02.049425 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:15:02.049383    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2c2494c1-9709-4ae8-9730-67eafc2d1425" path="/var/lib/kubelet/pods/2c2494c1-9709-4ae8-9730-67eafc2d1425/volumes"
Jul 13 20:15:07.587346 ip-10-0-128-215 kubenswrapper[2565]: E0713 20:15:07.587292    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" podUID="a4b3b3ac-0484-42d4-a875-f74db7379ea1"
Jul 13 20:15:08.231335 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:15:08.231300    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 13 20:15:14.406978 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:15:14.406950    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2bscc_2f0655bd-4a09-4314-8c1b-e30191eaec0e/ovn-acl-logging/0.log"
Jul 13 20:15:14.421337 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:15:14.421305    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2bscc_2f0655bd-4a09-4314-8c1b-e30191eaec0e/ovn-acl-logging/0.log"
Jul 13 20:15:32.718160 ip-10-0-128-215 kubenswrapper[2565]: E0713 20:15:32.718113    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dmww7f" podUID="634b78c4-751c-4880-9f6c-1ea69b737987"
Jul 13 20:15:33.352853 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:15:33.352815    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dmww7f"
Jul 13 20:16:37.061448 ip-10-0-128-215 kubenswrapper[2565]: E0713 20:16:37.061395    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8gwt" podUID="74e7b5d4-9d6e-4736-98c3-09d48ed6dec2"
Jul 13 20:16:37.061981 ip-10-0-128-215 kubenswrapper[2565]: E0713 20:16:37.061395    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mhsdz" podUID="762ad0f9-19a6-4a36-b4eb-a50f7291ab13"
Jul 13 20:16:37.667573 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:16:37.667534    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mhsdz"
Jul 13 20:16:37.667815 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:16:37.667537    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8gwt"
Jul 13 20:17:11.232350 ip-10-0-128-215 kubenswrapper[2565]: E0713 20:17:11.232247    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" podUID="a4b3b3ac-0484-42d4-a875-f74db7379ea1"
Jul 13 20:17:11.847328 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:17:11.847290    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 13 20:17:36.354261 ip-10-0-128-215 kubenswrapper[2565]: E0713 20:17:36.354211    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dmww7f" podUID="634b78c4-751c-4880-9f6c-1ea69b737987"
Jul 13 20:17:36.986565 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:17:36.986529    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dmww7f"
Jul 13 20:18:40.669094 ip-10-0-128-215 kubenswrapper[2565]: E0713 20:18:40.669008    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mhsdz" podUID="762ad0f9-19a6-4a36-b4eb-a50f7291ab13"
Jul 13 20:18:40.669094 ip-10-0-128-215 kubenswrapper[2565]: E0713 20:18:40.669062    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8gwt" podUID="74e7b5d4-9d6e-4736-98c3-09d48ed6dec2"
Jul 13 20:18:41.295388 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:18:41.295358    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8gwt"
Jul 13 20:18:41.295578 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:18:41.295359    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mhsdz"
Jul 13 20:19:14.848628 ip-10-0-128-215 kubenswrapper[2565]: E0713 20:19:14.848586    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" podUID="a4b3b3ac-0484-42d4-a875-f74db7379ea1"
Jul 13 20:19:15.455643 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:19:15.455611    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 13 20:19:39.988087 ip-10-0-128-215 kubenswrapper[2565]: E0713 20:19:39.988037    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dmww7f" podUID="634b78c4-751c-4880-9f6c-1ea69b737987"
Jul 13 20:19:40.569842 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:19:40.569809    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dmww7f"
Jul 13 20:20:14.440304 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:20:14.440274    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2bscc_2f0655bd-4a09-4314-8c1b-e30191eaec0e/ovn-acl-logging/0.log"
Jul 13 20:20:14.454468 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:20:14.454445    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2bscc_2f0655bd-4a09-4314-8c1b-e30191eaec0e/ovn-acl-logging/0.log"
Jul 13 20:20:44.296370 ip-10-0-128-215 kubenswrapper[2565]: E0713 20:20:44.296331    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mhsdz" podUID="762ad0f9-19a6-4a36-b4eb-a50f7291ab13"
Jul 13 20:20:44.296370 ip-10-0-128-215 kubenswrapper[2565]: E0713 20:20:44.296331    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8gwt" podUID="74e7b5d4-9d6e-4736-98c3-09d48ed6dec2"
Jul 13 20:20:44.888192 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:20:44.888154    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8gwt"
Jul 13 20:20:44.888405 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:20:44.888161    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mhsdz"
Jul 13 20:21:18.456335 ip-10-0-128-215 kubenswrapper[2565]: E0713 20:21:18.456297    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" podUID="a4b3b3ac-0484-42d4-a875-f74db7379ea1"
Jul 13 20:21:19.052884 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:21:19.052856    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 13 20:21:43.570801 ip-10-0-128-215 kubenswrapper[2565]: E0713 20:21:43.570685    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dmww7f" podUID="634b78c4-751c-4880-9f6c-1ea69b737987"
Jul 13 20:21:44.183133 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:21:44.183100    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dmww7f"
Jul 13 20:22:47.889294 ip-10-0-128-215 kubenswrapper[2565]: E0713 20:22:47.889237    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mhsdz" podUID="762ad0f9-19a6-4a36-b4eb-a50f7291ab13"
Jul 13 20:22:47.889294 ip-10-0-128-215 kubenswrapper[2565]: E0713 20:22:47.889295    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8gwt" podUID="74e7b5d4-9d6e-4736-98c3-09d48ed6dec2"
Jul 13 20:22:48.485195 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:22:48.485157    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8gwt"
Jul 13 20:22:48.485388 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:22:48.485169    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mhsdz"
Jul 13 20:23:11.948456 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:23:11.948376    2565 ???:1] "http: TLS handshake error from 10.0.131.203:51746: EOF"
Jul 13 20:23:12.010346 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:23:12.010315    2565 ???:1] "http: TLS handshake error from 10.0.128.215:52190: EOF"
Jul 13 20:23:22.053609 ip-10-0-128-215 kubenswrapper[2565]: E0713 20:23:22.053564    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" podUID="a4b3b3ac-0484-42d4-a875-f74db7379ea1"
Jul 13 20:23:22.646896 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:23:22.646865    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 13 20:23:41.505035 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:23:41.505002    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-655f7c58b4-8g97n_87a48a7a-6d63-43f0-8bab-7e29c1a6979a/istio-proxy/1.log"
Jul 13 20:23:41.516565 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:23:41.516538    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-655f7c58b4-8g97n_87a48a7a-6d63-43f0-8bab-7e29c1a6979a/istio-proxy/2.log"
Jul 13 20:23:43.147769 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:23:43.147737    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-655f7c58b4-8g97n_87a48a7a-6d63-43f0-8bab-7e29c1a6979a/istio-proxy/1.log"
Jul 13 20:23:43.159978 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:23:43.159949    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-655f7c58b4-8g97n_87a48a7a-6d63-43f0-8bab-7e29c1a6979a/istio-proxy/2.log"
Jul 13 20:23:44.848577 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:23:44.848547    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-655f7c58b4-8g97n_87a48a7a-6d63-43f0-8bab-7e29c1a6979a/istio-proxy/1.log"
Jul 13 20:23:44.862453 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:23:44.862435    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-655f7c58b4-8g97n_87a48a7a-6d63-43f0-8bab-7e29c1a6979a/istio-proxy/2.log"
Jul 13 20:23:46.450446 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:23:46.450414    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-655f7c58b4-8g97n_87a48a7a-6d63-43f0-8bab-7e29c1a6979a/istio-proxy/1.log"
Jul 13 20:23:46.463416 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:23:46.463393    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-655f7c58b4-8g97n_87a48a7a-6d63-43f0-8bab-7e29c1a6979a/istio-proxy/2.log"
Jul 13 20:23:47.184587 ip-10-0-128-215 kubenswrapper[2565]: E0713 20:23:47.184527    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dmww7f" podUID="634b78c4-751c-4880-9f6c-1ea69b737987"
Jul 13 20:23:47.554722 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:23:47.554670    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-85fd4b8b9d-4tc92_6b37279a-c1b7-4e01-b19a-b6b69bf6be78/kuadrant-console-plugin/0.log"
Jul 13 20:23:47.765062 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:23:47.765028    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dmww7f"
Jul 13 20:23:47.828753 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:23:47.828656    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-69574b596d-hnvp6_bee0c256-af28-4777-89d9-411b937b62fc/limitador/0.log"
Jul 13 20:23:49.030614 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:23:49.030579    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-85fd4b8b9d-4tc92_6b37279a-c1b7-4e01-b19a-b6b69bf6be78/kuadrant-console-plugin/0.log"
Jul 13 20:23:49.314379 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:23:49.314303    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-69574b596d-hnvp6_bee0c256-af28-4777-89d9-411b937b62fc/limitador/0.log"
Jul 13 20:23:50.550976 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:23:50.550948    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-85fd4b8b9d-4tc92_6b37279a-c1b7-4e01-b19a-b6b69bf6be78/kuadrant-console-plugin/0.log"
Jul 13 20:23:50.822389 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:23:50.822303    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-69574b596d-hnvp6_bee0c256-af28-4777-89d9-411b937b62fc/limitador/0.log"
Jul 13 20:23:51.958021 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:23:51.957988    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-85fd4b8b9d-4tc92_6b37279a-c1b7-4e01-b19a-b6b69bf6be78/kuadrant-console-plugin/0.log"
Jul 13 20:23:52.245617 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:23:52.245531    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-69574b596d-hnvp6_bee0c256-af28-4777-89d9-411b937b62fc/limitador/0.log"
Jul 13 20:23:53.404524 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:23:53.404491    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-85fd4b8b9d-4tc92_6b37279a-c1b7-4e01-b19a-b6b69bf6be78/kuadrant-console-plugin/0.log"
Jul 13 20:23:53.690210 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:23:53.690126    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-69574b596d-hnvp6_bee0c256-af28-4777-89d9-411b937b62fc/limitador/0.log"
Jul 13 20:23:54.842979 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:23:54.842954    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-85fd4b8b9d-4tc92_6b37279a-c1b7-4e01-b19a-b6b69bf6be78/kuadrant-console-plugin/0.log"
Jul 13 20:23:55.110826 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:23:55.110736    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-69574b596d-hnvp6_bee0c256-af28-4777-89d9-411b937b62fc/limitador/0.log"
Jul 13 20:23:56.305389 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:23:56.305360    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-85fd4b8b9d-4tc92_6b37279a-c1b7-4e01-b19a-b6b69bf6be78/kuadrant-console-plugin/0.log"
Jul 13 20:23:56.592833 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:23:56.592759    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-69574b596d-hnvp6_bee0c256-af28-4777-89d9-411b937b62fc/limitador/0.log"
Jul 13 20:23:57.750785 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:23:57.750754    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-85fd4b8b9d-4tc92_6b37279a-c1b7-4e01-b19a-b6b69bf6be78/kuadrant-console-plugin/0.log"
Jul 13 20:23:58.025303 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:23:58.025278    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-69574b596d-hnvp6_bee0c256-af28-4777-89d9-411b937b62fc/limitador/0.log"
Jul 13 20:23:59.203087 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:23:59.203056    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-85fd4b8b9d-4tc92_6b37279a-c1b7-4e01-b19a-b6b69bf6be78/kuadrant-console-plugin/0.log"
Jul 13 20:23:59.484897 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:23:59.484819    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-69574b596d-hnvp6_bee0c256-af28-4777-89d9-411b937b62fc/limitador/0.log"
Jul 13 20:24:00.630471 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:24:00.630440    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-85fd4b8b9d-4tc92_6b37279a-c1b7-4e01-b19a-b6b69bf6be78/kuadrant-console-plugin/0.log"
Jul 13 20:24:00.924482 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:24:00.924405    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-69574b596d-hnvp6_bee0c256-af28-4777-89d9-411b937b62fc/limitador/0.log"
Jul 13 20:24:02.114745 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:24:02.114694    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-85fd4b8b9d-4tc92_6b37279a-c1b7-4e01-b19a-b6b69bf6be78/kuadrant-console-plugin/0.log"
Jul 13 20:24:02.357273 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:24:02.357238    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-69574b596d-hnvp6_bee0c256-af28-4777-89d9-411b937b62fc/limitador/0.log"
Jul 13 20:24:03.555077 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:24:03.555050    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-85fd4b8b9d-4tc92_6b37279a-c1b7-4e01-b19a-b6b69bf6be78/kuadrant-console-plugin/0.log"
Jul 13 20:24:03.858358 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:24:03.858275    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-69574b596d-hnvp6_bee0c256-af28-4777-89d9-411b937b62fc/limitador/0.log"
Jul 13 20:24:05.077854 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:24:05.077825    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-85fd4b8b9d-4tc92_6b37279a-c1b7-4e01-b19a-b6b69bf6be78/kuadrant-console-plugin/0.log"
Jul 13 20:24:05.401637 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:24:05.401556    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-69574b596d-hnvp6_bee0c256-af28-4777-89d9-411b937b62fc/limitador/0.log"
Jul 13 20:24:06.697092 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:24:06.697065    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-85fd4b8b9d-4tc92_6b37279a-c1b7-4e01-b19a-b6b69bf6be78/kuadrant-console-plugin/0.log"
Jul 13 20:24:06.977573 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:24:06.977494    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-69574b596d-hnvp6_bee0c256-af28-4777-89d9-411b937b62fc/limitador/0.log"
Jul 13 20:24:08.121941 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:24:08.121911    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-85fd4b8b9d-4tc92_6b37279a-c1b7-4e01-b19a-b6b69bf6be78/kuadrant-console-plugin/0.log"
Jul 13 20:24:08.393116 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:24:08.393030    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-69574b596d-hnvp6_bee0c256-af28-4777-89d9-411b937b62fc/limitador/0.log"
Jul 13 20:24:09.591481 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:24:09.591455    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-85fd4b8b9d-4tc92_6b37279a-c1b7-4e01-b19a-b6b69bf6be78/kuadrant-console-plugin/0.log"
Jul 13 20:24:09.917668 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:24:09.917581    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-69574b596d-hnvp6_bee0c256-af28-4777-89d9-411b937b62fc/limitador/0.log"
Jul 13 20:24:11.143219 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:24:11.143190    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-85fd4b8b9d-4tc92_6b37279a-c1b7-4e01-b19a-b6b69bf6be78/kuadrant-console-plugin/0.log"
Jul 13 20:24:11.408308 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:24:11.408229    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-69574b596d-hnvp6_bee0c256-af28-4777-89d9-411b937b62fc/limitador/0.log"
Jul 13 20:24:12.625260 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:24:12.625224    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-85fd4b8b9d-4tc92_6b37279a-c1b7-4e01-b19a-b6b69bf6be78/kuadrant-console-plugin/0.log"
Jul 13 20:24:12.914948 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:24:12.914870    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-69574b596d-hnvp6_bee0c256-af28-4777-89d9-411b937b62fc/limitador/0.log"
Jul 13 20:24:14.140995 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:24:14.140964    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-85fd4b8b9d-4tc92_6b37279a-c1b7-4e01-b19a-b6b69bf6be78/kuadrant-console-plugin/0.log"
Jul 13 20:24:14.514690 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:24:14.514660    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-69574b596d-hnvp6_bee0c256-af28-4777-89d9-411b937b62fc/limitador/0.log"
Jul 13 20:24:15.696820 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:24:15.696774    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-85fd4b8b9d-4tc92_6b37279a-c1b7-4e01-b19a-b6b69bf6be78/kuadrant-console-plugin/0.log"
Jul 13 20:24:15.970738 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:24:15.970607    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-69574b596d-hnvp6_bee0c256-af28-4777-89d9-411b937b62fc/limitador/0.log"
Jul 13 20:24:17.235285 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:24:17.235255    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-85fd4b8b9d-4tc92_6b37279a-c1b7-4e01-b19a-b6b69bf6be78/kuadrant-console-plugin/0.log"
Jul 13 20:24:17.562465 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:24:17.562440    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-69574b596d-hnvp6_bee0c256-af28-4777-89d9-411b937b62fc/limitador/0.log"
Jul 13 20:24:18.759429 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:24:18.759397    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-85fd4b8b9d-4tc92_6b37279a-c1b7-4e01-b19a-b6b69bf6be78/kuadrant-console-plugin/0.log"
Jul 13 20:24:19.050963 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:24:19.050934    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-69574b596d-hnvp6_bee0c256-af28-4777-89d9-411b937b62fc/limitador/0.log"
Jul 13 20:24:20.188887 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:24:20.188856    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-85fd4b8b9d-4tc92_6b37279a-c1b7-4e01-b19a-b6b69bf6be78/kuadrant-console-plugin/0.log"
Jul 13 20:24:20.507676 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:24:20.507594    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-69574b596d-hnvp6_bee0c256-af28-4777-89d9-411b937b62fc/limitador/0.log"
Jul 13 20:24:21.755762 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:24:21.755733    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-85fd4b8b9d-4tc92_6b37279a-c1b7-4e01-b19a-b6b69bf6be78/kuadrant-console-plugin/0.log"
Jul 13 20:24:22.121539 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:24:22.121507    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-69574b596d-hnvp6_bee0c256-af28-4777-89d9-411b937b62fc/limitador/0.log"
Jul 13 20:24:23.464250 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:24:23.464219    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-85fd4b8b9d-4tc92_6b37279a-c1b7-4e01-b19a-b6b69bf6be78/kuadrant-console-plugin/0.log"
Jul 13 20:24:23.742539 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:24:23.742437    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-69574b596d-hnvp6_bee0c256-af28-4777-89d9-411b937b62fc/limitador/0.log"
Jul 13 20:24:25.038323 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:24:25.038290    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-85fd4b8b9d-4tc92_6b37279a-c1b7-4e01-b19a-b6b69bf6be78/kuadrant-console-plugin/0.log"
Jul 13 20:24:25.355560 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:24:25.355463    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-69574b596d-hnvp6_bee0c256-af28-4777-89d9-411b937b62fc/limitador/0.log"
Jul 13 20:24:26.565091 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:24:26.565062    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-85fd4b8b9d-4tc92_6b37279a-c1b7-4e01-b19a-b6b69bf6be78/kuadrant-console-plugin/0.log"
Jul 13 20:24:26.859327 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:24:26.859228    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-69574b596d-hnvp6_bee0c256-af28-4777-89d9-411b937b62fc/limitador/0.log"
Jul 13 20:24:28.024410 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:24:28.024379    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-85fd4b8b9d-4tc92_6b37279a-c1b7-4e01-b19a-b6b69bf6be78/kuadrant-console-plugin/0.log"
Jul 13 20:24:28.308677 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:24:28.308589    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-69574b596d-hnvp6_bee0c256-af28-4777-89d9-411b937b62fc/limitador/0.log"
Jul 13 20:24:29.551236 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:24:29.551209    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-85fd4b8b9d-4tc92_6b37279a-c1b7-4e01-b19a-b6b69bf6be78/kuadrant-console-plugin/0.log"
Jul 13 20:24:29.847749 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:24:29.847641    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-69574b596d-hnvp6_bee0c256-af28-4777-89d9-411b937b62fc/limitador/0.log"
Jul 13 20:24:31.081553 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:24:31.081525    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-85fd4b8b9d-4tc92_6b37279a-c1b7-4e01-b19a-b6b69bf6be78/kuadrant-console-plugin/0.log"
Jul 13 20:24:31.418856 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:24:31.418755    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-69574b596d-hnvp6_bee0c256-af28-4777-89d9-411b937b62fc/limitador/0.log"
Jul 13 20:24:32.855392 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:24:32.855361    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-85fd4b8b9d-4tc92_6b37279a-c1b7-4e01-b19a-b6b69bf6be78/kuadrant-console-plugin/0.log"
Jul 13 20:24:33.164312 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:24:33.164218    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-69574b596d-hnvp6_bee0c256-af28-4777-89d9-411b937b62fc/limitador/0.log"
Jul 13 20:24:39.679647 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:24:39.679556    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-qlzr5_6bbc0748-e539-4679-a100-1ba45794129b/global-pull-secret-syncer/0.log"
Jul 13 20:24:39.892501 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:24:39.892466    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-n4lpd_9dccc391-865f-45a2-9004-85e3830f39be/konnectivity-agent/0.log"
Jul 13 20:24:40.148671 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:24:40.148637    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-128-215.ec2.internal_33d859ca02a99923eded717ac20e75f3/haproxy/0.log"
Jul 13 20:24:45.200600 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:24:45.200571    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-85fd4b8b9d-4tc92_6b37279a-c1b7-4e01-b19a-b6b69bf6be78/kuadrant-console-plugin/0.log"
Jul 13 20:24:45.553731 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:24:45.553684    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-69574b596d-hnvp6_bee0c256-af28-4777-89d9-411b937b62fc/limitador/0.log"
Jul 13 20:24:46.998127 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:24:46.998092    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a22aa65f-fb55-49ec-a10e-c14a494422de/alertmanager/0.log"
Jul 13 20:24:47.108946 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:24:47.108911    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a22aa65f-fb55-49ec-a10e-c14a494422de/config-reloader/0.log"
Jul 13 20:24:47.233731 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:24:47.233678    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a22aa65f-fb55-49ec-a10e-c14a494422de/kube-rbac-proxy-web/0.log"
Jul 13 20:24:47.511847 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:24:47.511818    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a22aa65f-fb55-49ec-a10e-c14a494422de/kube-rbac-proxy/0.log"
Jul 13 20:24:47.588186 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:24:47.588155    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a22aa65f-fb55-49ec-a10e-c14a494422de/kube-rbac-proxy-metric/0.log"
Jul 13 20:24:47.650369 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:24:47.650334    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a22aa65f-fb55-49ec-a10e-c14a494422de/prom-label-proxy/0.log"
Jul 13 20:24:47.713827 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:24:47.713795    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a22aa65f-fb55-49ec-a10e-c14a494422de/init-config-reloader/0.log"
Jul 13 20:24:47.821120 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:24:47.821019    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-5f7f494dd4-whnln_ae45caa3-cc5b-4100-809e-2961c7fab65c/cluster-monitoring-operator/0.log"
Jul 13 20:24:48.306476 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:24:48.306447    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5f68df7f58-jmgtf_ab3e7251-ebd4-487c-bef3-ecdffa077837/monitoring-plugin/0.log"
Jul 13 20:24:48.583744 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:24:48.583647    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-s5sd7_d984d1bc-572a-4897-a32b-af3f919735f2/node-exporter/0.log"
Jul 13 20:24:48.633948 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:24:48.633911    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-s5sd7_d984d1bc-572a-4897-a32b-af3f919735f2/kube-rbac-proxy/0.log"
Jul 13 20:24:48.679148 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:24:48.679118    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-s5sd7_d984d1bc-572a-4897-a32b-af3f919735f2/init-textfile/0.log"
Jul 13 20:24:49.292641 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:24:49.292598    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-64785b6675-dbsvm_5e935794-9a93-4a39-b78d-1e973574ad31/prometheus-operator/0.log"
Jul 13 20:24:49.418995 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:24:49.418971    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-64785b6675-dbsvm_5e935794-9a93-4a39-b78d-1e973574ad31/kube-rbac-proxy/0.log"
Jul 13 20:24:49.522835 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:24:49.522808    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5699db478d-mjnwl_1c30602a-28cb-45df-9400-9e4d295cde13/telemeter-client/0.log"
Jul 13 20:24:49.583507 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:24:49.583424    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-s6wcw/perf-node-gather-daemonset-767rd"]
Jul 13 20:24:49.583873 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:24:49.583859    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c2494c1-9709-4ae8-9730-67eafc2d1425" containerName="storage-initializer"
Jul 13 20:24:49.583933 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:24:49.583876    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c2494c1-9709-4ae8-9730-67eafc2d1425" containerName="storage-initializer"
Jul 13 20:24:49.583933 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:24:49.583883    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c2494c1-9709-4ae8-9730-67eafc2d1425" containerName="main"
Jul 13 20:24:49.583933 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:24:49.583889    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c2494c1-9709-4ae8-9730-67eafc2d1425" containerName="main"
Jul 13 20:24:49.584029 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:24:49.583959    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c2494c1-9709-4ae8-9730-67eafc2d1425" containerName="main"
Jul 13 20:24:49.587210 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:24:49.587187    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-s6wcw/perf-node-gather-daemonset-767rd"
Jul 13 20:24:49.597661 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:24:49.597636    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-s6wcw\"/\"default-dockercfg-g9b4n\""
Jul 13 20:24:49.598661 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:24:49.598641    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-s6wcw\"/\"kube-root-ca.crt\""
Jul 13 20:24:49.598774 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:24:49.598682    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-s6wcw\"/\"openshift-service-ca.crt\""
Jul 13 20:24:49.616380 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:24:49.616341    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-s6wcw/perf-node-gather-daemonset-767rd"]
Jul 13 20:24:49.651293 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:24:49.651258    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5699db478d-mjnwl_1c30602a-28cb-45df-9400-9e4d295cde13/reload/0.log"
Jul 13 20:24:49.688057 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:24:49.688013    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/628b1923-7fab-485a-a9c8-626671027d24-proc\") pod \"perf-node-gather-daemonset-767rd\" (UID: \"628b1923-7fab-485a-a9c8-626671027d24\") " pod="openshift-must-gather-s6wcw/perf-node-gather-daemonset-767rd"
Jul 13 20:24:49.688057 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:24:49.688059    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62dvn\" (UniqueName: \"kubernetes.io/projected/628b1923-7fab-485a-a9c8-626671027d24-kube-api-access-62dvn\") pod \"perf-node-gather-daemonset-767rd\" (UID: \"628b1923-7fab-485a-a9c8-626671027d24\") " pod="openshift-must-gather-s6wcw/perf-node-gather-daemonset-767rd"
Jul 13 20:24:49.688300 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:24:49.688085    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/628b1923-7fab-485a-a9c8-626671027d24-lib-modules\") pod \"perf-node-gather-daemonset-767rd\" (UID: \"628b1923-7fab-485a-a9c8-626671027d24\") " pod="openshift-must-gather-s6wcw/perf-node-gather-daemonset-767rd"
Jul 13 20:24:49.688300 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:24:49.688200    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/628b1923-7fab-485a-a9c8-626671027d24-podres\") pod \"perf-node-gather-daemonset-767rd\" (UID: \"628b1923-7fab-485a-a9c8-626671027d24\") " pod="openshift-must-gather-s6wcw/perf-node-gather-daemonset-767rd"
Jul 13 20:24:49.688300 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:24:49.688289    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/628b1923-7fab-485a-a9c8-626671027d24-sys\") pod \"perf-node-gather-daemonset-767rd\" (UID: \"628b1923-7fab-485a-a9c8-626671027d24\") " pod="openshift-must-gather-s6wcw/perf-node-gather-daemonset-767rd"
Jul 13 20:24:49.717565 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:24:49.717531    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5699db478d-mjnwl_1c30602a-28cb-45df-9400-9e4d295cde13/kube-rbac-proxy/0.log"
Jul 13 20:24:49.765951 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:24:49.765909    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5d9d8df5d-6dwcd_52addd15-9664-48bb-81a2-a2c683c03624/thanos-query/0.log"
Jul 13 20:24:49.789102 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:24:49.789065    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/628b1923-7fab-485a-a9c8-626671027d24-podres\") pod \"perf-node-gather-daemonset-767rd\" (UID: \"628b1923-7fab-485a-a9c8-626671027d24\") " pod="openshift-must-gather-s6wcw/perf-node-gather-daemonset-767rd"
Jul 13 20:24:49.789233 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:24:49.789154    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/628b1923-7fab-485a-a9c8-626671027d24-sys\") pod \"perf-node-gather-daemonset-767rd\" (UID: \"628b1923-7fab-485a-a9c8-626671027d24\") " pod="openshift-must-gather-s6wcw/perf-node-gather-daemonset-767rd"
Jul 13 20:24:49.789233 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:24:49.789222    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/628b1923-7fab-485a-a9c8-626671027d24-proc\") pod \"perf-node-gather-daemonset-767rd\" (UID: \"628b1923-7fab-485a-a9c8-626671027d24\") " pod="openshift-must-gather-s6wcw/perf-node-gather-daemonset-767rd"
Jul 13 20:24:49.789307 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:24:49.789251    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62dvn\" (UniqueName: \"kubernetes.io/projected/628b1923-7fab-485a-a9c8-626671027d24-kube-api-access-62dvn\") pod \"perf-node-gather-daemonset-767rd\" (UID: \"628b1923-7fab-485a-a9c8-626671027d24\") " pod="openshift-must-gather-s6wcw/perf-node-gather-daemonset-767rd"
Jul 13 20:24:49.789307 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:24:49.789259    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/628b1923-7fab-485a-a9c8-626671027d24-podres\") pod \"perf-node-gather-daemonset-767rd\" (UID: \"628b1923-7fab-485a-a9c8-626671027d24\") " pod="openshift-must-gather-s6wcw/perf-node-gather-daemonset-767rd"
Jul 13 20:24:49.789307 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:24:49.789283    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/628b1923-7fab-485a-a9c8-626671027d24-sys\") pod \"perf-node-gather-daemonset-767rd\" (UID: \"628b1923-7fab-485a-a9c8-626671027d24\") " pod="openshift-must-gather-s6wcw/perf-node-gather-daemonset-767rd"
Jul 13 20:24:49.789307 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:24:49.789288    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/628b1923-7fab-485a-a9c8-626671027d24-lib-modules\") pod \"perf-node-gather-daemonset-767rd\" (UID: \"628b1923-7fab-485a-a9c8-626671027d24\") " pod="openshift-must-gather-s6wcw/perf-node-gather-daemonset-767rd"
Jul 13 20:24:49.789433 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:24:49.789313    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/628b1923-7fab-485a-a9c8-626671027d24-proc\") pod \"perf-node-gather-daemonset-767rd\" (UID: \"628b1923-7fab-485a-a9c8-626671027d24\") " pod="openshift-must-gather-s6wcw/perf-node-gather-daemonset-767rd"
Jul 13 20:24:49.789433 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:24:49.789382    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/628b1923-7fab-485a-a9c8-626671027d24-lib-modules\") pod \"perf-node-gather-daemonset-767rd\" (UID: \"628b1923-7fab-485a-a9c8-626671027d24\") " pod="openshift-must-gather-s6wcw/perf-node-gather-daemonset-767rd"
Jul 13 20:24:49.800122 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:24:49.800092    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-62dvn\" (UniqueName: \"kubernetes.io/projected/628b1923-7fab-485a-a9c8-626671027d24-kube-api-access-62dvn\") pod \"perf-node-gather-daemonset-767rd\" (UID: \"628b1923-7fab-485a-a9c8-626671027d24\") " pod="openshift-must-gather-s6wcw/perf-node-gather-daemonset-767rd"
Jul 13 20:24:49.802903 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:24:49.802874    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5d9d8df5d-6dwcd_52addd15-9664-48bb-81a2-a2c683c03624/kube-rbac-proxy-web/0.log"
Jul 13 20:24:49.842033 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:24:49.841950    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5d9d8df5d-6dwcd_52addd15-9664-48bb-81a2-a2c683c03624/kube-rbac-proxy/0.log"
Jul 13 20:24:49.884870 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:24:49.884841    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5d9d8df5d-6dwcd_52addd15-9664-48bb-81a2-a2c683c03624/prom-label-proxy/0.log"
Jul 13 20:24:49.898141 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:24:49.898104    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-s6wcw/perf-node-gather-daemonset-767rd"
Jul 13 20:24:49.932468 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:24:49.932443    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5d9d8df5d-6dwcd_52addd15-9664-48bb-81a2-a2c683c03624/kube-rbac-proxy-rules/0.log"
Jul 13 20:24:49.967278 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:24:49.967239    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5d9d8df5d-6dwcd_52addd15-9664-48bb-81a2-a2c683c03624/kube-rbac-proxy-metrics/0.log"
Jul 13 20:24:50.055716 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:24:50.055675    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-s6wcw/perf-node-gather-daemonset-767rd"]
Jul 13 20:24:50.058063 ip-10-0-128-215 kubenswrapper[2565]: W0713 20:24:50.058033    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod628b1923_7fab_485a_a9c8_626671027d24.slice/crio-6c18422f1cf74f9af3a39ac19d315df456bf63055c0ac8bfea2fe420ceec181a WatchSource:0}: Error finding container 6c18422f1cf74f9af3a39ac19d315df456bf63055c0ac8bfea2fe420ceec181a: Status 404 returned error can't find the container with id 6c18422f1cf74f9af3a39ac19d315df456bf63055c0ac8bfea2fe420ceec181a
Jul 13 20:24:50.077437 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:24:50.077402    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-s6wcw/perf-node-gather-daemonset-767rd" event={"ID":"628b1923-7fab-485a-a9c8-626671027d24","Type":"ContainerStarted","Data":"6c18422f1cf74f9af3a39ac19d315df456bf63055c0ac8bfea2fe420ceec181a"}
Jul 13 20:24:51.084475 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:24:51.084425    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-s6wcw/perf-node-gather-daemonset-767rd" event={"ID":"628b1923-7fab-485a-a9c8-626671027d24","Type":"ContainerStarted","Data":"45cfa4c131bec03a9279cd9f02974c0a16bb798744ae0271800eaaaac65822d3"}
Jul 13 20:24:51.084936 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:24:51.084503    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-s6wcw/perf-node-gather-daemonset-767rd"
Jul 13 20:24:51.085576 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:24:51.085552    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-6cf856758c-bdj79_a10721d0-fc1e-485e-848b-7b877ad97903/networking-console-plugin/0.log"
Jul 13 20:24:51.485999 ip-10-0-128-215 kubenswrapper[2565]: E0713 20:24:51.485890    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mhsdz" podUID="762ad0f9-19a6-4a36-b4eb-a50f7291ab13"
Jul 13 20:24:51.485999 ip-10-0-128-215 kubenswrapper[2565]: E0713 20:24:51.485909    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8gwt" podUID="74e7b5d4-9d6e-4736-98c3-09d48ed6dec2"
Jul 13 20:24:52.088689 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:24:52.088648    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-mhsdz"
Jul 13 20:24:52.089235 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:24:52.088885    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79j8gwt"
Jul 13 20:24:52.188657 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:24:52.188626    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-8bbbbb749-r68gb_4f5c11d7-4b13-401a-bbec-9b9516fd5cda/console/0.log"
Jul 13 20:24:52.842300 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:24:52.842270    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-78495597f6-vds4p_85363100-f0a6-463d-9c43-abf1539636f8/volume-data-source-validator/0.log"
Jul 13 20:24:53.985436 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:24:53.985404    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-jw7lc_5a144761-c470-4ac9-95cb-a629c9aac64a/dns/0.log"
Jul 13 20:24:54.015262 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:24:54.015236    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-jw7lc_5a144761-c470-4ac9-95cb-a629c9aac64a/kube-rbac-proxy/0.log"
Jul 13 20:24:54.212469 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:24:54.212439    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-zcq5p_6ae58b1e-3aaf-45dc-a741-2fb895468ca7/dns-node-resolver/0.log"
Jul 13 20:24:54.779936 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:24:54.779896    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-7d57676984-24sqj_8a5ea70c-0be6-434d-81a5-7c72ac3bd61f/registry/0.log"
Jul 13 20:24:54.848637 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:24:54.848586    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-k2vg4_b7240a5a-c704-49a7-b152-bdeb36cee9d4/node-ca/0.log"
Jul 13 20:24:56.566576 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:24:56.566536    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-655f7c58b4-8g97n_87a48a7a-6d63-43f0-8bab-7e29c1a6979a/istio-proxy/1.log"
Jul 13 20:24:56.579468 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:24:56.579438    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-655f7c58b4-8g97n_87a48a7a-6d63-43f0-8bab-7e29c1a6979a/istio-proxy/2.log"
Jul 13 20:24:57.099876 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:24:57.099839    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-s6wcw/perf-node-gather-daemonset-767rd"
Jul 13 20:24:57.122979 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:24:57.122919    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-s6wcw/perf-node-gather-daemonset-767rd" podStartSLOduration=8.122900026 podStartE2EDuration="8.122900026s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 20:24:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 20:24:51.118434272 +0000 UTC m=+5677.643567711" watchObservedRunningTime="2026-07-13 20:24:57.122900026 +0000 UTC m=+5683.648033442"
Jul 13 20:24:57.160536 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:24:57.160505    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-4rt5g_2ddf2ee1-c9d7-4f30-aa52-b94c6d250c70/serve-healthcheck-canary/0.log"
Jul 13 20:24:58.011898 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:24:58.011867    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-xmzjq_053446da-3c51-46b2-a759-aad264c7b0f9/kube-rbac-proxy/0.log"
Jul 13 20:24:58.036760 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:24:58.036728    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-xmzjq_053446da-3c51-46b2-a759-aad264c7b0f9/exporter/0.log"
Jul 13 20:24:58.067693 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:24:58.067666    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-xmzjq_053446da-3c51-46b2-a759-aad264c7b0f9/extractor/0.log"
Jul 13 20:25:01.484921 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:25:01.484809    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-5ff584b858-xkf9w_18724f8f-9798-46b5-a6a1-574ab45fcb20/manager/0.log"
Jul 13 20:25:01.670248 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:25:01.670211    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-655f7c58b4-8g97n_87a48a7a-6d63-43f0-8bab-7e29c1a6979a/istio-proxy/1.log"
Jul 13 20:25:01.682921 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:25:01.682887    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-655f7c58b4-8g97n_87a48a7a-6d63-43f0-8bab-7e29c1a6979a/istio-proxy/2.log"
Jul 13 20:25:03.566791 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:25:03.566727    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-655f7c58b4-8g97n_87a48a7a-6d63-43f0-8bab-7e29c1a6979a/istio-proxy/1.log"
Jul 13 20:25:03.582024 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:25:03.581983    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-655f7c58b4-8g97n_87a48a7a-6d63-43f0-8bab-7e29c1a6979a/istio-proxy/2.log"
Jul 13 20:25:03.589566 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:25:03.589532    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-ctct5_60a3ab50-3f54-418e-8b39-4bbd8b5d92bf/s3-init/0.log"
Jul 13 20:25:03.611859 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:25:03.611825    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-ctct5_60a3ab50-3f54-418e-8b39-4bbd8b5d92bf/download-hf-model/0.log"
Jul 13 20:25:03.650451 ip-10-0-128-215 kubenswrapper[2565]: I0713 20:25:03.650424    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-64568bcd49-5tglb_e2a6c28c-da28-464e-8636-4f5b1cafcefc/seaweedfs/0.log"