May 26 11:24:36.972517 ip-10-0-136-194 systemd[1]: Starting Kubernetes Kubelet...
May 26 11:24:37.423152 ip-10-0-136-194 kubenswrapper[2572]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
May 26 11:24:37.423152 ip-10-0-136-194 kubenswrapper[2572]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
May 26 11:24:37.423152 ip-10-0-136-194 kubenswrapper[2572]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
May 26 11:24:37.423152 ip-10-0-136-194 kubenswrapper[2572]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
May 26 11:24:37.423152 ip-10-0-136-194 kubenswrapper[2572]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
May 26 11:24:37.425073 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.424982    2572 server.go:212] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
May 26 11:24:37.428894 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.428879    2572 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 26 11:24:37.428894 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.428894    2572 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 26 11:24:37.428967 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.428898    2572 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 26 11:24:37.428967 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.428901    2572 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 26 11:24:37.428967 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.428905    2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 26 11:24:37.428967 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.428908    2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 26 11:24:37.428967 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.428910    2572 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 26 11:24:37.428967 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.428913    2572 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 26 11:24:37.428967 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.428917    2572 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 26 11:24:37.428967 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.428919    2572 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 26 11:24:37.428967 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.428922    2572 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 26 11:24:37.428967 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.428924    2572 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 26 11:24:37.428967 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.428927    2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 26 11:24:37.428967 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.428929    2572 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 26 11:24:37.428967 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.428932    2572 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 26 11:24:37.428967 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.428935    2572 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 26 11:24:37.428967 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.428938    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 26 11:24:37.428967 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.428940    2572 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 26 11:24:37.428967 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.428943    2572 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 26 11:24:37.428967 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.428951    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 26 11:24:37.428967 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.428954    2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 26 11:24:37.428967 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.428957    2572 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 26 11:24:37.429469 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.428959    2572 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 26 11:24:37.429469 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.428962    2572 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 26 11:24:37.429469 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.428965    2572 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 26 11:24:37.429469 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.428967    2572 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 26 11:24:37.429469 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.428970    2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 26 11:24:37.429469 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.428973    2572 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 26 11:24:37.429469 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.428976    2572 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 26 11:24:37.429469 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.428979    2572 feature_gate.go:328] unrecognized feature gate: Example2
May 26 11:24:37.429469 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.428981    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 26 11:24:37.429469 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.428984    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 26 11:24:37.429469 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.428987    2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 26 11:24:37.429469 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.428990    2572 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 26 11:24:37.429469 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.428992    2572 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 26 11:24:37.429469 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.428995    2572 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 26 11:24:37.429469 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.428998    2572 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 26 11:24:37.429469 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.429000    2572 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 26 11:24:37.429469 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.429003    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 26 11:24:37.429469 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.429006    2572 feature_gate.go:328] unrecognized feature gate: DualReplica
May 26 11:24:37.429469 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.429008    2572 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 26 11:24:37.429469 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.429013    2572 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 26 11:24:37.429966 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.429017    2572 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 26 11:24:37.429966 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.429022    2572 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 26 11:24:37.429966 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.429025    2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 26 11:24:37.429966 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.429028    2572 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 26 11:24:37.429966 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.429031    2572 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 26 11:24:37.429966 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.429034    2572 feature_gate.go:328] unrecognized feature gate: Example
May 26 11:24:37.429966 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.429036    2572 feature_gate.go:328] unrecognized feature gate: NewOLM
May 26 11:24:37.429966 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.429039    2572 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 26 11:24:37.429966 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.429042    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 26 11:24:37.429966 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.429044    2572 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 26 11:24:37.429966 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.429047    2572 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 26 11:24:37.429966 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.429049    2572 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 26 11:24:37.429966 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.429052    2572 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 26 11:24:37.429966 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.429055    2572 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 26 11:24:37.429966 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.429058    2572 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 26 11:24:37.429966 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.429061    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 26 11:24:37.429966 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.429063    2572 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 26 11:24:37.429966 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.429066    2572 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 26 11:24:37.429966 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.429068    2572 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 26 11:24:37.430451 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.429072    2572 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 26 11:24:37.430451 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.429074    2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 26 11:24:37.430451 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.429077    2572 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 26 11:24:37.430451 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.429079    2572 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 26 11:24:37.430451 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.429082    2572 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 26 11:24:37.430451 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.429084    2572 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 26 11:24:37.430451 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.429087    2572 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 26 11:24:37.430451 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.429091    2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 26 11:24:37.430451 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.429093    2572 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 26 11:24:37.430451 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.429096    2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 26 11:24:37.430451 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.429098    2572 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 26 11:24:37.430451 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.429100    2572 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 26 11:24:37.430451 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.429103    2572 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 26 11:24:37.430451 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.429106    2572 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 26 11:24:37.430451 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.429108    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 26 11:24:37.430451 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.429110    2572 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 26 11:24:37.430451 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.429113    2572 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 26 11:24:37.430451 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.429116    2572 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 26 11:24:37.430451 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.429118    2572 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 26 11:24:37.430451 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.429122    2572 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 26 11:24:37.431018 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.429125    2572 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 26 11:24:37.431018 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.429127    2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 26 11:24:37.431018 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.429130    2572 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 26 11:24:37.431018 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.429132    2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 26 11:24:37.431018 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.429135    2572 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 26 11:24:37.431018 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.429512    2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 26 11:24:37.431018 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.429517    2572 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 26 11:24:37.431018 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.429520    2572 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 26 11:24:37.431018 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.429523    2572 feature_gate.go:328] unrecognized feature gate: Example2
May 26 11:24:37.431018 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.429526    2572 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 26 11:24:37.431018 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.429529    2572 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 26 11:24:37.431018 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.429532    2572 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 26 11:24:37.431018 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.429535    2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 26 11:24:37.431018 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.429538    2572 feature_gate.go:328] unrecognized feature gate: DualReplica
May 26 11:24:37.431018 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.429540    2572 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 26 11:24:37.431018 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.429544    2572 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 26 11:24:37.431018 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.429547    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 26 11:24:37.431018 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.429550    2572 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 26 11:24:37.431018 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.429553    2572 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 26 11:24:37.431018 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.429555    2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 26 11:24:37.431525 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.429558    2572 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 26 11:24:37.431525 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.429561    2572 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 26 11:24:37.431525 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.429563    2572 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 26 11:24:37.431525 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.429568    2572 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 26 11:24:37.431525 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.429571    2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 26 11:24:37.431525 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.429575    2572 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 26 11:24:37.431525 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.429577    2572 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 26 11:24:37.431525 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.429580    2572 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 26 11:24:37.431525 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.429582    2572 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 26 11:24:37.431525 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.429585    2572 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 26 11:24:37.431525 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.429587    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 26 11:24:37.431525 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.429590    2572 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 26 11:24:37.431525 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.429592    2572 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 26 11:24:37.431525 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.429595    2572 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 26 11:24:37.431525 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.429598    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 26 11:24:37.431525 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.429600    2572 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 26 11:24:37.431525 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.429602    2572 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 26 11:24:37.431525 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.429606    2572 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 26 11:24:37.431525 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.429610    2572 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 26 11:24:37.432007 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.429613    2572 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 26 11:24:37.432007 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.429615    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 26 11:24:37.432007 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.429618    2572 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 26 11:24:37.432007 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.429620    2572 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 26 11:24:37.432007 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.429623    2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 26 11:24:37.432007 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.429625    2572 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 26 11:24:37.432007 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.429628    2572 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 26 11:24:37.432007 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.429631    2572 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 26 11:24:37.432007 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.429634    2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 26 11:24:37.432007 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.429636    2572 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 26 11:24:37.432007 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.429639    2572 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 26 11:24:37.432007 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.429641    2572 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 26 11:24:37.432007 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.429644    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 26 11:24:37.432007 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.429647    2572 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 26 11:24:37.432007 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.429649    2572 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 26 11:24:37.432007 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.429652    2572 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 26 11:24:37.432007 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.429654    2572 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 26 11:24:37.432007 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.429660    2572 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 26 11:24:37.432007 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.429662    2572 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 26 11:24:37.432007 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.429665    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 26 11:24:37.432528 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.429668    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 26 11:24:37.432528 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.429670    2572 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 26 11:24:37.432528 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.429673    2572 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 26 11:24:37.432528 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.429676    2572 feature_gate.go:328] unrecognized feature gate: Example
May 26 11:24:37.432528 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.429678    2572 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 26 11:24:37.432528 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.429681    2572 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 26 11:24:37.432528 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.429683    2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 26 11:24:37.432528 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.429686    2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 26 11:24:37.432528 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.429688    2572 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 26 11:24:37.432528 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.429691    2572 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 26 11:24:37.432528 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.429693    2572 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 26 11:24:37.432528 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.429698    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 26 11:24:37.432528 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.429700    2572 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 26 11:24:37.432528 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.429704    2572 feature_gate.go:328] unrecognized feature gate: NewOLM
May 26 11:24:37.432528 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.429707    2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 26 11:24:37.432528 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.429709    2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 26 11:24:37.432528 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.429712    2572 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 26 11:24:37.432528 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.429715    2572 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 26 11:24:37.432528 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.429717    2572 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 26 11:24:37.432528 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.429720    2572 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 26 11:24:37.433026 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.429723    2572 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 26 11:24:37.433026 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.429726    2572 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 26 11:24:37.433026 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.429728    2572 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 26 11:24:37.433026 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.429731    2572 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 26 11:24:37.433026 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.429734    2572 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 26 11:24:37.433026 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.429736    2572 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 26 11:24:37.433026 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.429739    2572 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 26 11:24:37.433026 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.429741    2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 26 11:24:37.433026 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.429744    2572 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 26 11:24:37.433026 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.429746    2572 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 26 11:24:37.433026 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.429749    2572 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 26 11:24:37.433026 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.429752    2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 26 11:24:37.433026 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.431045    2572 flags.go:64] FLAG: --address="0.0.0.0"
May 26 11:24:37.433026 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.431055    2572 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
May 26 11:24:37.433026 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.431066    2572 flags.go:64] FLAG: --anonymous-auth="true"
May 26 11:24:37.433026 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.431071    2572 flags.go:64] FLAG: --application-metrics-count-limit="100"
May 26 11:24:37.433026 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.431075    2572 flags.go:64] FLAG: --authentication-token-webhook="false"
May 26 11:24:37.433026 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.431079    2572 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
May 26 11:24:37.433026 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.431083    2572 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
May 26 11:24:37.433026 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.431088    2572 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
May 26 11:24:37.433026 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.431091    2572 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
May 26 11:24:37.433570 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.431094    2572 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
May 26 11:24:37.433570 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.431097    2572 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
May 26 11:24:37.433570 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.431101    2572 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
May 26 11:24:37.433570 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.431104    2572 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
May 26 11:24:37.433570 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.431107    2572 flags.go:64] FLAG: --cgroup-root=""
May 26 11:24:37.433570 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.431110    2572 flags.go:64] FLAG: --cgroups-per-qos="true"
May 26 11:24:37.433570 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.431113    2572 flags.go:64] FLAG: --client-ca-file=""
May 26 11:24:37.433570 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.431116    2572 flags.go:64] FLAG: --cloud-config=""
May 26 11:24:37.433570 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.431119    2572 flags.go:64] FLAG: --cloud-provider="external"
May 26 11:24:37.433570 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.431122    2572 flags.go:64] FLAG: --cluster-dns="[]"
May 26 11:24:37.433570 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.431128    2572 flags.go:64] FLAG: --cluster-domain=""
May 26 11:24:37.433570 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.431131    2572 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
May 26 11:24:37.433570 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.431134    2572 flags.go:64] FLAG: --config-dir=""
May 26 11:24:37.433570 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.431137    2572 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
May 26 11:24:37.433570 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.431141    2572 flags.go:64] FLAG: --container-log-max-files="5"
May 26 11:24:37.433570 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.431145    2572 flags.go:64] FLAG: --container-log-max-size="10Mi"
May 26 11:24:37.433570 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.431148    2572 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
May 26 11:24:37.433570 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.431151    2572 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
May 26 11:24:37.433570 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.431154    2572 flags.go:64] FLAG: --containerd-namespace="k8s.io"
May 26 11:24:37.433570 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.431158    2572 flags.go:64] FLAG: --contention-profiling="false"
May 26 11:24:37.433570 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.431161    2572 flags.go:64] FLAG: --cpu-cfs-quota="true"
May 26 11:24:37.433570 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.431163    2572 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
May 26 11:24:37.433570 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.431167    2572 flags.go:64] FLAG: --cpu-manager-policy="none"
May 26 11:24:37.433570 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.431170    2572 flags.go:64] FLAG: --cpu-manager-policy-options=""
May 26 11:24:37.433570 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.431174    2572 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
May 26 11:24:37.434174 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.431177    2572 flags.go:64] FLAG: --enable-controller-attach-detach="true"
May 26 11:24:37.434174 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.431180    2572 flags.go:64] FLAG: --enable-debugging-handlers="true"
May 26 11:24:37.434174 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.431195    2572 flags.go:64] FLAG: --enable-load-reader="false"
May 26 11:24:37.434174 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.431198    2572 flags.go:64] FLAG: --enable-server="true"
May 26 11:24:37.434174 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.431201    2572 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
May 26 11:24:37.434174 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.431206    2572 flags.go:64] FLAG: --event-burst="100"
May 26 11:24:37.434174 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.431209    2572 flags.go:64] FLAG: --event-qps="50"
May 26 11:24:37.434174 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.431212    2572 flags.go:64] FLAG: --event-storage-age-limit="default=0"
May 26 11:24:37.434174 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.431215    2572 flags.go:64] FLAG: --event-storage-event-limit="default=0"
May 26 11:24:37.434174 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.431218    2572 flags.go:64] FLAG: --eviction-hard=""
May 26 11:24:37.434174 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.431222    2572 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
May 26 11:24:37.434174 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.431225    2572 flags.go:64] FLAG: --eviction-minimum-reclaim=""
May 26 11:24:37.434174 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.431228    2572 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
May 26 11:24:37.434174 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.431231    2572 flags.go:64] FLAG: --eviction-soft=""
May 26 11:24:37.434174 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.431234    2572 flags.go:64] FLAG: --eviction-soft-grace-period=""
May 26 11:24:37.434174 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.431237    2572 flags.go:64] FLAG: --exit-on-lock-contention="false"
May 26 11:24:37.434174 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.431240    2572 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
May 26 11:24:37.434174 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.431243    2572 flags.go:64] FLAG: --experimental-mounter-path=""
May 26 11:24:37.434174 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.431246    2572 flags.go:64] FLAG: --fail-cgroupv1="false"
May 26 11:24:37.434174 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.431249    2572 flags.go:64] FLAG: --fail-swap-on="true"
May 26 11:24:37.434174 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.431252    2572 flags.go:64] FLAG: --feature-gates=""
May 26 11:24:37.434174 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.431255    2572 flags.go:64] FLAG: --file-check-frequency="20s"
May 26 11:24:37.434174 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.431258    2572 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
May 26 11:24:37.434174 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.431262    2572 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
May 26 11:24:37.434174 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.431265    2572 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
May 26 11:24:37.434831 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.431268    2572 flags.go:64] FLAG: --healthz-port="10248"
May 26 11:24:37.434831 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.431271    2572 flags.go:64] FLAG: --help="false"
May 26 11:24:37.434831 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.431274    2572 flags.go:64] FLAG: --hostname-override="ip-10-0-136-194.ec2.internal"
May 26 11:24:37.434831 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.431277    2572 flags.go:64] FLAG: --housekeeping-interval="10s"
May 26 11:24:37.434831 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.431280    2572 flags.go:64] FLAG: --http-check-frequency="20s"
May 26 11:24:37.434831 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.431283    2572 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
May 26 11:24:37.434831 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.431287    2572 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
May 26 11:24:37.434831 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.431290    2572 flags.go:64] FLAG: --image-gc-high-threshold="85"
May 26 11:24:37.434831 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.431293    2572 flags.go:64] FLAG: --image-gc-low-threshold="80"
May 26 11:24:37.434831 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.431296    2572 flags.go:64] FLAG: --image-service-endpoint=""
May 26 11:24:37.434831 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.431299    2572 flags.go:64] FLAG: --kernel-memcg-notification="false"
May 26 11:24:37.434831 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.431302    2572 flags.go:64] FLAG: --kube-api-burst="100"
May 26 11:24:37.434831 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.431305    2572 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
May 26 11:24:37.434831 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.431308    2572 flags.go:64] FLAG: --kube-api-qps="50"
May 26 11:24:37.434831 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.431311    2572 flags.go:64] FLAG: --kube-reserved=""
May 26 11:24:37.434831 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.431315    2572 flags.go:64] FLAG: --kube-reserved-cgroup=""
May 26 11:24:37.434831 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.431318    2572 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
May 26 11:24:37.434831 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.431321    2572 flags.go:64] FLAG: --kubelet-cgroups=""
May 26 11:24:37.434831 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.431324    2572 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
May 26 11:24:37.434831 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.431327    2572 flags.go:64] FLAG: --lock-file=""
May 26 11:24:37.434831 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.431330    2572 flags.go:64] FLAG: --log-cadvisor-usage="false"
May 26 11:24:37.434831 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.431333    2572 flags.go:64] FLAG: --log-flush-frequency="5s"
May 26 11:24:37.434831 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.431336    2572 flags.go:64] FLAG: --log-json-info-buffer-size="0"
May 26 11:24:37.434831 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.431341    2572 flags.go:64] FLAG: --log-json-split-stream="false"
May 26 11:24:37.435400 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.431344    2572 flags.go:64] FLAG: --log-text-info-buffer-size="0"
May 26 11:24:37.435400 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.431347    2572 flags.go:64] FLAG: --log-text-split-stream="false"
May 26 11:24:37.435400 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.431350    2572 flags.go:64] FLAG: --logging-format="text"
May 26 11:24:37.435400 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.431352    2572 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
May 26 11:24:37.435400 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.431356    2572 flags.go:64] FLAG: --make-iptables-util-chains="true"
May 26 11:24:37.435400 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.431359    2572 flags.go:64] FLAG: --manifest-url=""
May 26 11:24:37.435400 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.431361    2572 flags.go:64] FLAG: --manifest-url-header=""
May 26 11:24:37.435400 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.431366    2572 flags.go:64] FLAG: --max-housekeeping-interval="15s"
May 26 11:24:37.435400 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.431369    2572 flags.go:64] FLAG: --max-open-files="1000000"
May 26 11:24:37.435400 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.431373    2572 flags.go:64] FLAG: --max-pods="110"
May 26 11:24:37.435400 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.431376    2572 flags.go:64] FLAG: --maximum-dead-containers="-1"
May 26 11:24:37.435400 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.431379    2572 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
May 26 11:24:37.435400 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.431382    2572 flags.go:64] FLAG: --memory-manager-policy="None"
May 26 11:24:37.435400 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.431385    2572 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
May 26 11:24:37.435400 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.431389    2572 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
May 26 11:24:37.435400 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.431392    2572 flags.go:64] FLAG: --node-ip="0.0.0.0"
May 26 11:24:37.435400 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.431395    2572 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
May 26 11:24:37.435400 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.431403    2572 flags.go:64] FLAG: --node-status-max-images="50"
May 26 11:24:37.435400 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.431406    2572 flags.go:64] FLAG: --node-status-update-frequency="10s"
May 26 11:24:37.435400 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.431409    2572 flags.go:64] FLAG: --oom-score-adj="-999"
May 26 11:24:37.435400 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.431413    2572 flags.go:64] FLAG: --pod-cidr=""
May 26 11:24:37.435400 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.431415    2572 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:afe88671bf163c42b467d159dc6f0b784dbe28b1ed44e171edc9c06fd99cc6fc"
May 26 11:24:37.435400 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.431421    2572 flags.go:64] FLAG: --pod-manifest-path=""
May 26 11:24:37.435964 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.431425    2572 flags.go:64] FLAG: --pod-max-pids="-1"
May 26 11:24:37.435964 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.431428    2572 flags.go:64] FLAG: --pods-per-core="0"
May 26 11:24:37.435964 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.431431    2572 flags.go:64] FLAG: --port="10250"
May 26 11:24:37.435964 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.431435    2572 flags.go:64] FLAG: --protect-kernel-defaults="false"
May 26 11:24:37.435964 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.431438    2572 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-01bf730db4bf741e6"
May 26 11:24:37.435964 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.431441    2572 flags.go:64] FLAG: --qos-reserved=""
May 26 11:24:37.435964 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.431444    2572 flags.go:64] FLAG: --read-only-port="10255"
May 26 11:24:37.435964 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.431447    2572 flags.go:64] FLAG: --register-node="true"
May 26 11:24:37.435964 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.431450    2572 flags.go:64] FLAG: --register-schedulable="true"
May 26 11:24:37.435964 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.431453    2572 flags.go:64] FLAG: --register-with-taints=""
May 26 11:24:37.435964 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.431456    2572 flags.go:64] FLAG: --registry-burst="10"
May 26 11:24:37.435964 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.431459    2572 flags.go:64] FLAG: --registry-qps="5"
May 26 11:24:37.435964 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.431462    2572 flags.go:64] FLAG: --reserved-cpus=""
May 26 11:24:37.435964 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.431465    2572 flags.go:64] FLAG: --reserved-memory=""
May 26 11:24:37.435964 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.431469    2572 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
May 26 11:24:37.435964 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.431472    2572 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
May 26 11:24:37.435964 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.431475    2572 flags.go:64] FLAG: --rotate-certificates="false"
May 26 11:24:37.435964 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.431478    2572 flags.go:64] FLAG: --rotate-server-certificates="false"
May 26 11:24:37.435964 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.431480    2572 flags.go:64] FLAG: --runonce="false"
May 26 11:24:37.435964 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.431483    2572 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
May 26 11:24:37.435964 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.431487    2572 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
May 26 11:24:37.435964 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.431490    2572 flags.go:64] FLAG: --seccomp-default="false"
May 26 11:24:37.435964 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.431493    2572 flags.go:64] FLAG: --serialize-image-pulls="true"
May 26 11:24:37.435964 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.431496    2572 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
May 26 11:24:37.435964 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.431499    2572 flags.go:64] FLAG: --storage-driver-db="cadvisor"
May 26 11:24:37.435964 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.431502    2572 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
May 26 11:24:37.436633 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.431505    2572 flags.go:64] FLAG: --storage-driver-password="root"
May 26 11:24:37.436633 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.431509    2572 flags.go:64] FLAG: --storage-driver-secure="false"
May 26 11:24:37.436633 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.431512    2572 flags.go:64] FLAG: --storage-driver-table="stats"
May 26 11:24:37.436633 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.431515    2572 flags.go:64] FLAG: --storage-driver-user="root"
May 26 11:24:37.436633 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.431518    2572 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
May 26 11:24:37.436633 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.431521    2572 flags.go:64] FLAG: --sync-frequency="1m0s"
May 26 11:24:37.436633 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.431525    2572 flags.go:64] FLAG: --system-cgroups=""
May 26 11:24:37.436633 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.431528    2572 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
May 26 11:24:37.436633 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.431534    2572 flags.go:64] FLAG: --system-reserved-cgroup=""
May 26 11:24:37.436633 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.431537    2572 flags.go:64] FLAG: --tls-cert-file=""
May 26 11:24:37.436633 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.431539    2572 flags.go:64] FLAG: --tls-cipher-suites="[]"
May 26 11:24:37.436633 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.431544    2572 flags.go:64] FLAG: --tls-min-version=""
May 26 11:24:37.436633 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.431547    2572 flags.go:64] FLAG: --tls-private-key-file=""
May 26 11:24:37.436633 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.431550    2572 flags.go:64] FLAG: --topology-manager-policy="none"
May 26 11:24:37.436633 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.431553    2572 flags.go:64] FLAG: --topology-manager-policy-options=""
May 26 11:24:37.436633 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.431556    2572 flags.go:64] FLAG: --topology-manager-scope="container"
May 26 11:24:37.436633 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.431559    2572 flags.go:64] FLAG: --v="2"
May 26 11:24:37.436633 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.431564    2572 flags.go:64] FLAG: --version="false"
May 26 11:24:37.436633 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.431568    2572 flags.go:64] FLAG: --vmodule=""
May 26 11:24:37.436633 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.431572    2572 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
May 26 11:24:37.436633 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.431576    2572 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
May 26 11:24:37.436633 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.431670    2572 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 26 11:24:37.436633 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.431674    2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 26 11:24:37.436633 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.431677    2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 26 11:24:37.437235 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.431681    2572 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 26 11:24:37.437235 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.431683    2572 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 26 11:24:37.437235 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.431686    2572 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 26 11:24:37.437235 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.431689    2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 26 11:24:37.437235 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.431691    2572 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 26 11:24:37.437235 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.431694    2572 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 26 11:24:37.437235 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.431697    2572 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 26 11:24:37.437235 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.431701    2572 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 26 11:24:37.437235 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.431703    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 26 11:24:37.437235 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.431706    2572 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 26 11:24:37.437235 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.431709    2572 feature_gate.go:328] unrecognized feature gate: DualReplica
May 26 11:24:37.437235 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.431711    2572 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 26 11:24:37.437235 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.431714    2572 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 26 11:24:37.437235 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.431716    2572 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 26 11:24:37.437235 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.431719    2572 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 26 11:24:37.437235 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.431722    2572 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 26 11:24:37.437235 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.431725    2572 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 26 11:24:37.437235 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.431727    2572 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 26 11:24:37.437235 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.431730    2572 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 26 11:24:37.437235 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.431732    2572 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 26 11:24:37.437783 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.431736    2572 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 26 11:24:37.437783 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.431740    2572 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 26 11:24:37.437783 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.431742    2572 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 26 11:24:37.437783 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.431745    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 26 11:24:37.437783 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.431747    2572 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 26 11:24:37.437783 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.431750    2572 feature_gate.go:328] unrecognized feature gate: Example
May 26 11:24:37.437783 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.431753    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 26 11:24:37.437783 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.431755    2572 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 26 11:24:37.437783 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.431758    2572 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 26 11:24:37.437783 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.431761    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 26 11:24:37.437783 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.431763    2572 feature_gate.go:328] unrecognized feature gate: NewOLM
May 26 11:24:37.437783 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.431766    2572 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 26 11:24:37.437783 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.431768    2572 feature_gate.go:328] unrecognized feature gate: Example2
May 26 11:24:37.437783 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.431771    2572 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 26 11:24:37.437783 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.431773    2572 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 26 11:24:37.437783 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.431776    2572 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 26 11:24:37.437783 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.431778    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 26 11:24:37.437783 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.431781    2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 26 11:24:37.437783 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.431783    2572 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 26 11:24:37.437783 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.431786    2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 26 11:24:37.438297 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.431794    2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 26 11:24:37.438297 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.431797    2572 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 26 11:24:37.438297 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.431800    2572 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 26 11:24:37.438297 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.431803    2572 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 26 11:24:37.438297 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.431805    2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 26 11:24:37.438297 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.431808    2572 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 26 11:24:37.438297 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.431811    2572 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 26 11:24:37.438297 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.431814    2572 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 26 11:24:37.438297 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.431817    2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 26 11:24:37.438297 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.431819    2572 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 26 11:24:37.438297 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.431822    2572 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 26 11:24:37.438297 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.431825    2572 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 26 11:24:37.438297 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.431827    2572 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 26 11:24:37.438297 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.431830    2572 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 26 11:24:37.438297 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.431833    2572 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 26 11:24:37.438297 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.431835    2572 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 26 11:24:37.438297 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.431838    2572 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 26 11:24:37.438297 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.431840    2572 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 26 11:24:37.438297 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.431844    2572 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 26 11:24:37.438769 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.431846    2572 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 26 11:24:37.438769 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.431849    2572 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 26 11:24:37.438769 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.431851    2572 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 26 11:24:37.438769 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.431854    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 26 11:24:37.438769 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.431856    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 26 11:24:37.438769 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.431859    2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 26 11:24:37.438769 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.431862    2572 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 26 11:24:37.438769 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.431864    2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 26 11:24:37.438769 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.431867    2572 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 26 11:24:37.438769 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.431869    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 26 11:24:37.438769 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.431872    2572 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 26 11:24:37.438769 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.431875    2572 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 26 11:24:37.438769 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.431877    2572 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 26 11:24:37.438769 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.431881    2572 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 26 11:24:37.438769 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.431884    2572 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 26 11:24:37.438769 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.431887    2572 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 26 11:24:37.438769 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.431890    2572 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 26 11:24:37.438769 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.431892    2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 26 11:24:37.438769 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.431895    2572 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 26 11:24:37.438769 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.431899    2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 26 11:24:37.439266 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.431901    2572 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 26 11:24:37.439266 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.431904    2572 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 26 11:24:37.439266 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.431908    2572 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 26 11:24:37.439266 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.431912    2572 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 26 11:24:37.439266 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.432544    2572 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]}
May 26 11:24:37.439762 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.439646    2572 server.go:530] "Kubelet version" kubeletVersion="v1.33.11"
May 26 11:24:37.439794 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.439766    2572 server.go:532] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
May 26 11:24:37.439828 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.439819    2572 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 26 11:24:37.439828 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.439828    2572 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 26 11:24:37.439881 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.439831    2572 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 26 11:24:37.439881 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.439834    2572 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 26 11:24:37.439881 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.439838    2572 feature_gate.go:328] unrecognized feature gate: DualReplica
May 26 11:24:37.439881 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.439841    2572 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 26 11:24:37.439881 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.439846    2572 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 26 11:24:37.439881 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.439850    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 26 11:24:37.439881 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.439855    2572 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 26 11:24:37.439881 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.439859    2572 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 26 11:24:37.439881 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.439862    2572 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 26 11:24:37.439881 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.439865    2572 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 26 11:24:37.439881 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.439867    2572 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 26 11:24:37.439881 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.439870    2572 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 26 11:24:37.439881 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.439873    2572 feature_gate.go:328] unrecognized feature gate: Example
May 26 11:24:37.439881 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.439876    2572 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 26 11:24:37.439881 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.439879    2572 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 26 11:24:37.439881 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.439881    2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 26 11:24:37.439881 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.439884    2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 26 11:24:37.439881 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.439887    2572 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 26 11:24:37.439881 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.439890    2572 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 26 11:24:37.440396 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.439893    2572 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 26 11:24:37.440396 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.439896    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 26 11:24:37.440396 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.439899    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 26 11:24:37.440396 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.439903    2572 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 26 11:24:37.440396 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.439907    2572 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 26 11:24:37.440396 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.439910    2572 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 26 11:24:37.440396 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.439912    2572 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 26 11:24:37.440396 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.439915    2572 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 26 11:24:37.440396 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.439917    2572 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 26 11:24:37.440396 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.439921    2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 26 11:24:37.440396 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.439927    2572 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 26 11:24:37.440396 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.439932    2572 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 26 11:24:37.440396 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.439936    2572 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 26 11:24:37.440396 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.439939    2572 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 26 11:24:37.440396 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.439942    2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 26 11:24:37.440396 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.439944    2572 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 26 11:24:37.440396 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.439947    2572 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 26 11:24:37.440396 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.439949    2572 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 26 11:24:37.440396 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.439952    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 26 11:24:37.440396 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.439955    2572 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 26 11:24:37.440886 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.439957    2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 26 11:24:37.440886 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.439960    2572 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 26 11:24:37.440886 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.439962    2572 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 26 11:24:37.440886 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.439965    2572 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 26 11:24:37.440886 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.439967    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 26 11:24:37.440886 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.439970    2572 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 26 11:24:37.440886 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.439972    2572 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 26 11:24:37.440886 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.439975    2572 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 26 11:24:37.440886 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.439977    2572 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 26 11:24:37.440886 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.439980    2572 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 26 11:24:37.440886 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.439982    2572 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 26 11:24:37.440886 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.439985    2572 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 26 11:24:37.440886 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.439987    2572 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 26 11:24:37.440886 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.439991    2572 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 26 11:24:37.440886 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.439994    2572 feature_gate.go:328] unrecognized feature gate: NewOLM
May 26 11:24:37.440886 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.439996    2572 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 26 11:24:37.440886 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.440000    2572 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 26 11:24:37.440886 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.440004    2572 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 26 11:24:37.440886 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.440007    2572 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 26 11:24:37.440886 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.440012    2572 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 26 11:24:37.441387 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.440016    2572 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 26 11:24:37.441387 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.440019    2572 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 26 11:24:37.441387 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.440021    2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 26 11:24:37.441387 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.440024    2572 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 26 11:24:37.441387 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.440027    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 26 11:24:37.441387 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.440029    2572 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 26 11:24:37.441387 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.440032    2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 26 11:24:37.441387 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.440034    2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 26 11:24:37.441387 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.440037    2572 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 26 11:24:37.441387 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.440039    2572 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 26 11:24:37.441387 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.440042    2572 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 26 11:24:37.441387 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.440045    2572 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 26 11:24:37.441387 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.440047    2572 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 26 11:24:37.441387 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.440049    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 26 11:24:37.441387 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.440052    2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 26 11:24:37.441387 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.440055    2572 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 26 11:24:37.441387 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.440057    2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 26 11:24:37.441387 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.440059    2572 feature_gate.go:328] unrecognized feature gate: Example2
May 26 11:24:37.441387 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.440063    2572 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 26 11:24:37.441857 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.440067    2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 26 11:24:37.441857 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.440070    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 26 11:24:37.441857 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.440074    2572 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 26 11:24:37.441857 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.440076    2572 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 26 11:24:37.441857 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.440080    2572 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 26 11:24:37.441857 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.440084    2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 26 11:24:37.441857 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.440093    2572 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]}
May 26 11:24:37.441857 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.440208    2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 26 11:24:37.441857 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.440213    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 26 11:24:37.441857 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.440216    2572 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 26 11:24:37.441857 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.440219    2572 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 26 11:24:37.441857 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.440222    2572 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 26 11:24:37.441857 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.440225    2572 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 26 11:24:37.441857 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.440228    2572 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 26 11:24:37.441857 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.440230    2572 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 26 11:24:37.442257 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.440233    2572 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 26 11:24:37.442257 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.440236    2572 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 26 11:24:37.442257 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.440241    2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 26 11:24:37.442257 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.440246    2572 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 26 11:24:37.442257 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.440250    2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 26 11:24:37.442257 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.440254    2572 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 26 11:24:37.442257 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.440257    2572 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 26 11:24:37.442257 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.440260    2572 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 26 11:24:37.442257 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.440262    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 26 11:24:37.442257 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.440266    2572 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 26 11:24:37.442257 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.440268    2572 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 26 11:24:37.442257 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.440271    2572 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 26 11:24:37.442257 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.440274    2572 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 26 11:24:37.442257 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.440276    2572 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 26 11:24:37.442257 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.440279    2572 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 26 11:24:37.442257 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.440281    2572 feature_gate.go:328] unrecognized feature gate: DualReplica
May 26 11:24:37.442257 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.440284    2572 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 26 11:24:37.442257 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.440286    2572 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 26 11:24:37.442257 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.440289    2572 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 26 11:24:37.442257 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.440292    2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 26 11:24:37.442745 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.440295    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 26 11:24:37.442745 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.440297    2572 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 26 11:24:37.442745 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.440300    2572 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 26 11:24:37.442745 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.440302    2572 feature_gate.go:328] unrecognized feature gate: Example
May 26 11:24:37.442745 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.440305    2572 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 26 11:24:37.442745 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.440307    2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 26 11:24:37.442745 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.440309    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 26 11:24:37.442745 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.440312    2572 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 26 11:24:37.442745 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.440315    2572 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 26 11:24:37.442745 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.440318    2572 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 26 11:24:37.442745 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.440322    2572 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 26 11:24:37.442745 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.440326    2572 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 26 11:24:37.442745 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.440330    2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 26 11:24:37.442745 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.440335    2572 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 26 11:24:37.442745 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.440338    2572 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 26 11:24:37.442745 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.440343    2572 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 26 11:24:37.442745 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.440347    2572 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 26 11:24:37.442745 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.440350    2572 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 26 11:24:37.442745 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.440352    2572 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 26 11:24:37.442745 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.440355    2572 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 26 11:24:37.443237 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.440358    2572 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 26 11:24:37.443237 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.440360    2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 26 11:24:37.443237 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.440363    2572 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 26 11:24:37.443237 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.440365    2572 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 26 11:24:37.443237 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.440368    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 26 11:24:37.443237 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.440371    2572 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 26 11:24:37.443237 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.440373    2572 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 26 11:24:37.443237 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.440376    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 26 11:24:37.443237 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.440378    2572 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 26 11:24:37.443237 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.440381    2572 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 26 11:24:37.443237 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.440384    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 26 11:24:37.443237 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.440386    2572 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 26 11:24:37.443237 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.440389    2572 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 26 11:24:37.443237 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.440391    2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 26 11:24:37.443237 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.440394    2572 feature_gate.go:328] unrecognized feature gate: Example2
May 26 11:24:37.443237 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.440398    2572 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 26 11:24:37.443237 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.440402    2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 26 11:24:37.443237 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.440407    2572 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 26 11:24:37.443237 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.440411    2572 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 26 11:24:37.443727 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.440415    2572 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 26 11:24:37.443727 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.440417    2572 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 26 11:24:37.443727 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.440420    2572 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 26 11:24:37.443727 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.440423    2572 feature_gate.go:328] unrecognized feature gate: NewOLM
May 26 11:24:37.443727 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.440425    2572 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 26 11:24:37.443727 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.440428    2572 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 26 11:24:37.443727 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.440430    2572 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 26 11:24:37.443727 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.440433    2572 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 26 11:24:37.443727 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.440437    2572 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 26 11:24:37.443727 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.440441    2572 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 26 11:24:37.443727 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.440444    2572 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 26 11:24:37.443727 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.440446    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 26 11:24:37.443727 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.440449    2572 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 26 11:24:37.443727 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.440452    2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 26 11:24:37.443727 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.440454    2572 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 26 11:24:37.443727 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.440457    2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 26 11:24:37.443727 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.440460    2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 26 11:24:37.443727 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.440462    2572 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 26 11:24:37.443727 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:37.440465    2572 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 26 11:24:37.444260 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.440470    2572 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]}
May 26 11:24:37.444260 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.441105    2572 server.go:962] "Client rotation is on, will bootstrap in background"
May 26 11:24:37.444260 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.443253    2572 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
May 26 11:24:37.444861 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.444850    2572 server.go:1019] "Starting client certificate rotation"
May 26 11:24:37.444962 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.444946    2572 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
May 26 11:24:37.445000 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.444989    2572 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
May 26 11:24:37.473645 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.473625    2572 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
May 26 11:24:37.477154 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.477127    2572 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
May 26 11:24:37.494327 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.494306    2572 log.go:25] "Validated CRI v1 runtime API"
May 26 11:24:37.500052 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.500035    2572 log.go:25] "Validated CRI v1 image API"
May 26 11:24:37.501163 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.501144    2572 server.go:1452] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
May 26 11:24:37.503494 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.503469    2572 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 c03934bf-44dd-4a7c-bd33-2b3fa3338c02:/dev/nvme0n1p3 f226fd00-d7e6-4b08-a0fb-728832c24b67:/dev/nvme0n1p4]
May 26 11:24:37.503567 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.503493    2572 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
May 26 11:24:37.504086 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.504057    2572 reflector.go:430] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
May 26 11:24:37.507991 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.507883    2572 manager.go:217] Machine: {Timestamp:2026-05-26 11:24:37.507388604 +0000 UTC m=+0.409598923 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3164968 MemoryCapacity:33164500992 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec28cec21c228468f057546f88fa9d80 SystemUUID:ec28cec2-1c22-8468-f057-546f88fa9d80 BootID:6d8e62c0-a133-4659-a953-d1182978add5 Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6103040 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582250496 Type:vfs Inodes:4048401 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:8c:a8:a5:a4:a1 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:8c:a8:a5:a4:a1 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:7a:f7:17:20:78:e8 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164500992 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
May 26 11:24:37.507991 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.507986    2572 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
May 26 11:24:37.508098 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.508064    2572 manager.go:233] Version: {KernelVersion:5.14.0-570.114.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260512-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
May 26 11:24:37.511960 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.511934    2572 container_manager_linux.go:270] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
May 26 11:24:37.512104 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.511963    2572 container_manager_linux.go:275] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-136-194.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
May 26 11:24:37.512148 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.512114    2572 topology_manager.go:138] "Creating topology manager with none policy"
May 26 11:24:37.512148 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.512122    2572 container_manager_linux.go:306] "Creating device plugin manager"
May 26 11:24:37.512148 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.512130    2572 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
May 26 11:24:37.513633 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.513621    2572 server.go:72] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
May 26 11:24:37.515292 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.515281    2572 state_mem.go:36] "Initialized new in-memory state store"
May 26 11:24:37.515400 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.515391    2572 server.go:1267] "Using root directory" path="/var/lib/kubelet"
May 26 11:24:37.517735 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.517725    2572 kubelet.go:491] "Attempting to sync node with API server"
May 26 11:24:37.517780 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.517742    2572 kubelet.go:386] "Adding static pod path" path="/etc/kubernetes/manifests"
May 26 11:24:37.517780 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.517754    2572 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
May 26 11:24:37.517780 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.517766    2572 kubelet.go:397] "Adding apiserver pod source"
May 26 11:24:37.517780 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.517775    2572 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
May 26 11:24:37.519015 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.519003    2572 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
May 26 11:24:37.519061 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.519022    2572 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
May 26 11:24:37.521915 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.521898    2572 kuberuntime_manager.go:279] "Container runtime initialized" containerRuntime="cri-o" version="1.33.11-2.rhaos4.20.gitb2a8320.el9" apiVersion="v1"
May 26 11:24:37.523683 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.523670    2572 kubelet.go:953] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
May 26 11:24:37.524422 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.524411    2572 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
May 26 11:24:37.524463 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.524432    2572 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
May 26 11:24:37.524463 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.524440    2572 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
May 26 11:24:37.524463 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.524446    2572 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
May 26 11:24:37.524463 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.524455    2572 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
May 26 11:24:37.524463 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.524460    2572 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/secret"
May 26 11:24:37.524589 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.524470    2572 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
May 26 11:24:37.524589 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.524476    2572 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
May 26 11:24:37.524589 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.524484    2572 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/fc"
May 26 11:24:37.524589 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.524490    2572 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
May 26 11:24:37.524589 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.524505    2572 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/projected"
May 26 11:24:37.524728 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.524719    2572 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
May 26 11:24:37.524756 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.524749    2572 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/csi"
May 26 11:24:37.524786 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.524757    2572 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/image"
May 26 11:24:37.528102 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.528087    2572 watchdog_linux.go:99] "Systemd watchdog is not enabled"
May 26 11:24:37.528204 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.528125    2572 server.go:1295] "Started kubelet"
May 26 11:24:37.528261 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.528198    2572 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
May 26 11:24:37.528308 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.528259    2572 ratelimit.go:55] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
May 26 11:24:37.528354 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.528317    2572 server_v1.go:47] "podresources" method="list" useActivePods=true
May 26 11:24:37.529014 ip-10-0-136-194 systemd[1]: Started Kubernetes Kubelet.
May 26 11:24:37.530455 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.530433    2572 server.go:255] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
May 26 11:24:37.533470 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.533454    2572 server.go:317] "Adding debug handlers to kubelet server"
May 26 11:24:37.535730 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.535707    2572 csi_plugin.go:988] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-136-194.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
May 26 11:24:37.535814 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:24:37.535751    2572 reflector.go:200] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
May 26 11:24:37.535814 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:24:37.535800    2572 reflector.go:200] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-136-194.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
May 26 11:24:37.537159 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.537140    2572 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
May 26 11:24:37.537250 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.537159    2572 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
May 26 11:24:37.537250 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.537234    2572 volume_manager.go:295] "The desired_state_of_world populator starts"
May 26 11:24:37.537250 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.537244    2572 volume_manager.go:297] "Starting Kubelet Volume Manager"
May 26 11:24:37.537405 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.537264    2572 desired_state_of_world_populator.go:150] "Desired state populator starts to run"
May 26 11:24:37.537405 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.537335    2572 reconstruct.go:97] "Volume reconstruction finished"
May 26 11:24:37.537405 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.537344    2572 reconciler.go:26] "Reconciler: start to sync state"
May 26 11:24:37.537994 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:24:37.537975    2572 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-136-194.ec2.internal\" not found"
May 26 11:24:37.538132 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:24:37.537034    2572 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-194.ec2.internal.18b319d457317b6b  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-194.ec2.internal,UID:ip-10-0-136-194.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-136-194.ec2.internal,},FirstTimestamp:2026-05-26 11:24:37.528099691 +0000 UTC m=+0.430310012,LastTimestamp:2026-05-26 11:24:37.528099691 +0000 UTC m=+0.430310012,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-194.ec2.internal,}"
May 26 11:24:37.539884 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.539818    2572 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
May 26 11:24:37.539884 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.539834    2572 factory.go:55] Registering systemd factory
May 26 11:24:37.539884 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.539843    2572 factory.go:223] Registration of the systemd container factory successfully
May 26 11:24:37.540070 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.540053    2572 factory.go:153] Registering CRI-O factory
May 26 11:24:37.540126 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.540073    2572 factory.go:223] Registration of the crio container factory successfully
May 26 11:24:37.540126 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.540091    2572 factory.go:103] Registering Raw factory
May 26 11:24:37.540126 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.540104    2572 manager.go:1196] Started watching for new ooms in manager
May 26 11:24:37.540657 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.540477    2572 manager.go:319] Starting recovery of all containers
May 26 11:24:37.540910 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:24:37.540878    2572 kubelet.go:1618] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
May 26 11:24:37.550495 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:24:37.550464    2572 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-136-194.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
May 26 11:24:37.550495 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:24:37.550469    2572 reflector.go:200] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
May 26 11:24:37.550771 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.550757    2572 manager.go:324] Recovery completed
May 26 11:24:37.551079 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.551061    2572 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-9vgq9"
May 26 11:24:37.554681 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.554669    2572 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 26 11:24:37.556450 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.556436    2572 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-136-194.ec2.internal" event="NodeHasSufficientMemory"
May 26 11:24:37.556510 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.556464    2572 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-136-194.ec2.internal" event="NodeHasNoDiskPressure"
May 26 11:24:37.556510 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.556478    2572 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-136-194.ec2.internal" event="NodeHasSufficientPID"
May 26 11:24:37.557003 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.556985    2572 cpu_manager.go:222] "Starting CPU manager" policy="none"
May 26 11:24:37.557003 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.557002    2572 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
May 26 11:24:37.557107 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.557017    2572 state_mem.go:36] "Initialized new in-memory state store"
May 26 11:24:37.559019 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.559005    2572 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-9vgq9"
May 26 11:24:37.559100 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.559063    2572 policy_none.go:49] "None policy: Start"
May 26 11:24:37.559100 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.559075    2572 memory_manager.go:186] "Starting memorymanager" policy="None"
May 26 11:24:37.559100 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.559085    2572 state_mem.go:35] "Initializing new in-memory state store"
May 26 11:24:37.608601 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.599701    2572 manager.go:341] "Starting Device Plugin manager"
May 26 11:24:37.608601 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:24:37.599735    2572 manager.go:517] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
May 26 11:24:37.608601 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.599745    2572 server.go:85] "Starting device plugin registration server"
May 26 11:24:37.608601 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.600021    2572 eviction_manager.go:189] "Eviction manager: starting control loop"
May 26 11:24:37.608601 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.600053    2572 container_log_manager.go:189] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
May 26 11:24:37.608601 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.600148    2572 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
May 26 11:24:37.608601 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.600242    2572 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
May 26 11:24:37.608601 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.600251    2572 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
May 26 11:24:37.608601 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:24:37.601176    2572 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
May 26 11:24:37.608601 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:24:37.601228    2572 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-136-194.ec2.internal\" not found"
May 26 11:24:37.669055 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.669018    2572 kubelet_network_linux.go:49] "Initialized iptables rules." protocol="IPv4"
May 26 11:24:37.670307 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.670290    2572 kubelet_network_linux.go:49] "Initialized iptables rules." protocol="IPv6"
May 26 11:24:37.670307 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.670310    2572 status_manager.go:230] "Starting to sync pod status with apiserver"
May 26 11:24:37.670454 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.670327    2572 watchdog_linux.go:127] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
May 26 11:24:37.670454 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.670338    2572 kubelet.go:2451] "Starting kubelet main sync loop"
May 26 11:24:37.670454 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:24:37.670437    2572 kubelet.go:2475] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
May 26 11:24:37.672748 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.672729    2572 reflector.go:430] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
May 26 11:24:37.700849 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.700767    2572 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 26 11:24:37.701944 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.701911    2572 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-136-194.ec2.internal" event="NodeHasSufficientMemory"
May 26 11:24:37.702043 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.701961    2572 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-136-194.ec2.internal" event="NodeHasNoDiskPressure"
May 26 11:24:37.702043 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.701977    2572 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-136-194.ec2.internal" event="NodeHasSufficientPID"
May 26 11:24:37.702043 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.702001    2572 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-136-194.ec2.internal"
May 26 11:24:37.713175 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.713154    2572 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-136-194.ec2.internal"
May 26 11:24:37.713301 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:24:37.713179    2572 kubelet_node_status.go:597] "Error updating node status, will retry" err="error getting node \"ip-10-0-136-194.ec2.internal\": node \"ip-10-0-136-194.ec2.internal\" not found"
May 26 11:24:37.734042 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:24:37.734019    2572 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-136-194.ec2.internal\" not found"
May 26 11:24:37.770747 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.770724    2572 kubelet.go:2537] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-194.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-136-194.ec2.internal"]
May 26 11:24:37.770824 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.770787    2572 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 26 11:24:37.771653 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.771638    2572 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-136-194.ec2.internal" event="NodeHasSufficientMemory"
May 26 11:24:37.771718 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.771669    2572 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-136-194.ec2.internal" event="NodeHasNoDiskPressure"
May 26 11:24:37.771718 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.771684    2572 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-136-194.ec2.internal" event="NodeHasSufficientPID"
May 26 11:24:37.772951 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.772939    2572 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 26 11:24:37.773130 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.773116    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-194.ec2.internal"
May 26 11:24:37.773182 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.773149    2572 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 26 11:24:37.773650 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.773632    2572 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-136-194.ec2.internal" event="NodeHasSufficientMemory"
May 26 11:24:37.773747 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.773658    2572 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-136-194.ec2.internal" event="NodeHasNoDiskPressure"
May 26 11:24:37.773747 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.773667    2572 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-136-194.ec2.internal" event="NodeHasSufficientPID"
May 26 11:24:37.773747 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.773640    2572 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-136-194.ec2.internal" event="NodeHasSufficientMemory"
May 26 11:24:37.773747 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.773741    2572 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-136-194.ec2.internal" event="NodeHasNoDiskPressure"
May 26 11:24:37.773907 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.773756    2572 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-136-194.ec2.internal" event="NodeHasSufficientPID"
May 26 11:24:37.774823 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.774808    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-194.ec2.internal"
May 26 11:24:37.774905 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.774832    2572 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 26 11:24:37.775415 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.775399    2572 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-136-194.ec2.internal" event="NodeHasSufficientMemory"
May 26 11:24:37.775505 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.775425    2572 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-136-194.ec2.internal" event="NodeHasNoDiskPressure"
May 26 11:24:37.775505 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.775438    2572 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-136-194.ec2.internal" event="NodeHasSufficientPID"
May 26 11:24:37.797795 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:24:37.797771    2572 kubelet.go:3336] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-194.ec2.internal\" not found" node="ip-10-0-136-194.ec2.internal"
May 26 11:24:37.802309 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:24:37.802293    2572 kubelet.go:3336] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-194.ec2.internal\" not found" node="ip-10-0-136-194.ec2.internal"
May 26 11:24:37.834918 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:24:37.834896    2572 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-136-194.ec2.internal\" not found"
May 26 11:24:37.838731 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.838714    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4d5d5c010d8065f0e4964c93d9cb64bf-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-194.ec2.internal\" (UID: \"4d5d5c010d8065f0e4964c93d9cb64bf\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-194.ec2.internal"
May 26 11:24:37.838787 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.838741    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4d5d5c010d8065f0e4964c93d9cb64bf-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-194.ec2.internal\" (UID: \"4d5d5c010d8065f0e4964c93d9cb64bf\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-194.ec2.internal"
May 26 11:24:37.838787 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.838760    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4e9c0f443794a935b3209de8b5efc30d-config\") pod \"kube-apiserver-proxy-ip-10-0-136-194.ec2.internal\" (UID: \"4e9c0f443794a935b3209de8b5efc30d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-194.ec2.internal"
May 26 11:24:37.935075 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:24:37.935041    2572 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-136-194.ec2.internal\" not found"
May 26 11:24:37.939337 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.939318    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4d5d5c010d8065f0e4964c93d9cb64bf-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-194.ec2.internal\" (UID: \"4d5d5c010d8065f0e4964c93d9cb64bf\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-194.ec2.internal"
May 26 11:24:37.939402 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.939348    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4d5d5c010d8065f0e4964c93d9cb64bf-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-194.ec2.internal\" (UID: \"4d5d5c010d8065f0e4964c93d9cb64bf\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-194.ec2.internal"
May 26 11:24:37.939402 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.939368    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4e9c0f443794a935b3209de8b5efc30d-config\") pod \"kube-apiserver-proxy-ip-10-0-136-194.ec2.internal\" (UID: \"4e9c0f443794a935b3209de8b5efc30d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-194.ec2.internal"
May 26 11:24:37.939473 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.939413    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4d5d5c010d8065f0e4964c93d9cb64bf-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-194.ec2.internal\" (UID: \"4d5d5c010d8065f0e4964c93d9cb64bf\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-194.ec2.internal"
May 26 11:24:37.939473 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.939461    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4e9c0f443794a935b3209de8b5efc30d-config\") pod \"kube-apiserver-proxy-ip-10-0-136-194.ec2.internal\" (UID: \"4e9c0f443794a935b3209de8b5efc30d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-194.ec2.internal"
May 26 11:24:37.939541 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:37.939483    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4d5d5c010d8065f0e4964c93d9cb64bf-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-194.ec2.internal\" (UID: \"4d5d5c010d8065f0e4964c93d9cb64bf\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-194.ec2.internal"
May 26 11:24:38.035821 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:24:38.035735    2572 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-136-194.ec2.internal\" not found"
May 26 11:24:38.100223 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:38.100200    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-194.ec2.internal"
May 26 11:24:38.104988 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:38.104971    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-194.ec2.internal"
May 26 11:24:38.136525 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:24:38.136458    2572 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-136-194.ec2.internal\" not found"
May 26 11:24:38.236887 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:24:38.236855    2572 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-136-194.ec2.internal\" not found"
May 26 11:24:38.337341 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:24:38.337250    2572 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-136-194.ec2.internal\" not found"
May 26 11:24:38.437722 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:24:38.437687    2572 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-136-194.ec2.internal\" not found"
May 26 11:24:38.444990 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:38.444964    2572 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
May 26 11:24:38.445128 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:38.445113    2572 reflector.go:556] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
May 26 11:24:38.538102 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:38.538075    2572 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
May 26 11:24:38.538102 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:24:38.538083    2572 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-136-194.ec2.internal\" not found"
May 26 11:24:38.555458 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:38.555431    2572 reflector.go:430] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
May 26 11:24:38.561301 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:38.561275    2572 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-05-25 11:19:37 +0000 UTC" deadline="2027-11-02 14:38:40.605007296 +0000 UTC"
May 26 11:24:38.561301 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:38.561299    2572 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12603h14m2.043710871s"
May 26 11:24:38.572647 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:38.572627    2572 reflector.go:430] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
May 26 11:24:38.582076 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:38.582055    2572 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-9c9vd"
May 26 11:24:38.588709 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:38.588649    2572 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-9c9vd"
May 26 11:24:38.630670 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:38.630622    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d5d5c010d8065f0e4964c93d9cb64bf.slice/crio-6fffa2e2f16d8d4d999db0cdbb134e9fbbcafe6f302b452da8eec01bd56ad746 WatchSource:0}: Error finding container 6fffa2e2f16d8d4d999db0cdbb134e9fbbcafe6f302b452da8eec01bd56ad746: Status 404 returned error can't find the container with id 6fffa2e2f16d8d4d999db0cdbb134e9fbbcafe6f302b452da8eec01bd56ad746
May 26 11:24:38.630938 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:38.630922    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e9c0f443794a935b3209de8b5efc30d.slice/crio-19df050d4d5c9eaac640fcecbb074219173938e463109f7cbe103b5ee3a76e02 WatchSource:0}: Error finding container 19df050d4d5c9eaac640fcecbb074219173938e463109f7cbe103b5ee3a76e02: Status 404 returned error can't find the container with id 19df050d4d5c9eaac640fcecbb074219173938e463109f7cbe103b5ee3a76e02
May 26 11:24:38.634288 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:38.634271    2572 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 26 11:24:38.638278 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:24:38.638255    2572 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-136-194.ec2.internal\" not found"
May 26 11:24:38.673689 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:38.673637    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-194.ec2.internal" event={"ID":"4d5d5c010d8065f0e4964c93d9cb64bf","Type":"ContainerStarted","Data":"6fffa2e2f16d8d4d999db0cdbb134e9fbbcafe6f302b452da8eec01bd56ad746"}
May 26 11:24:38.674615 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:38.674591    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-194.ec2.internal" event={"ID":"4e9c0f443794a935b3209de8b5efc30d","Type":"ContainerStarted","Data":"19df050d4d5c9eaac640fcecbb074219173938e463109f7cbe103b5ee3a76e02"}
May 26 11:24:38.725702 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:38.724551    2572 reflector.go:430] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
May 26 11:24:38.737633 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:38.737595    2572 kubelet.go:3340] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-194.ec2.internal"
May 26 11:24:38.752208 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:38.752165    2572 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
May 26 11:24:38.753310 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:38.753296    2572 kubelet.go:3340] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-194.ec2.internal"
May 26 11:24:38.762324 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:38.762308    2572 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
May 26 11:24:38.935386 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:38.935115    2572 reflector.go:430] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
May 26 11:24:39.317731 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.317651    2572 reflector.go:430] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
May 26 11:24:39.519011 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.518977    2572 apiserver.go:52] "Watching apiserver"
May 26 11:24:39.527458 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.527436    2572 reflector.go:430] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
May 26 11:24:39.529837 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.529812    2572 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-cluster-node-tuning-operator/tuned-qvkjq","openshift-image-registry/node-ca-s9s2b","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-194.ec2.internal","openshift-network-operator/iptables-alerter-gzwdn","kube-system/kube-apiserver-proxy-ip-10-0-136-194.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6g66v","openshift-dns/node-resolver-4clzr","openshift-multus/multus-additional-cni-plugins-vjbr5","openshift-multus/multus-fq6hb","openshift-multus/network-metrics-daemon-k7jgt","openshift-network-diagnostics/network-check-target-cc5xb","openshift-ovn-kubernetes/ovnkube-node-js8cc","kube-system/konnectivity-agent-n4h7j"]
May 26 11:24:39.532381 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.532357    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-4clzr"
May 26 11:24:39.534442 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.534421    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-s9s2b"
May 26 11:24:39.535234 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.535167    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-vlcw4\""
May 26 11:24:39.535553 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.535536    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
May 26 11:24:39.535638 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.535550    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
May 26 11:24:39.536592 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.536460    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-gzwdn"
May 26 11:24:39.536886 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.536860    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
May 26 11:24:39.537225 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.537206    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
May 26 11:24:39.537467 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.537442    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
May 26 11:24:39.538574 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.538553    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-qvkjq"
May 26 11:24:39.539639 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.539622    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
May 26 11:24:39.539735 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.539720    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
May 26 11:24:39.539735 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.539725    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
May 26 11:24:39.539873 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.539738    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-78lvm\""
May 26 11:24:39.539933 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.539896    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-7gwwf\""
May 26 11:24:39.540742 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.540724    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-vjbr5"
May 26 11:24:39.541736 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.541593    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-fpn7p\""
May 26 11:24:39.541806 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.541761    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
May 26 11:24:39.542872 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.542853    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k7jgt"
May 26 11:24:39.542961 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:24:39.542941    2572 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k7jgt" podUID="dede2966-ea61-4174-b667-c6fb46152d91"
May 26 11:24:39.545023 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.545006    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cc5xb"
May 26 11:24:39.545092 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:24:39.545056    2572 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cc5xb" podUID="e87b53a4-1dca-46b1-bbd4-e484919093c4"
May 26 11:24:39.545628 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.545438    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
May 26 11:24:39.545628 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.545594    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
May 26 11:24:39.545775 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.545744    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
May 26 11:24:39.545775 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.545751    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
May 26 11:24:39.545874 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.545802    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
May 26 11:24:39.545874 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.545754    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-sttsk\""
May 26 11:24:39.545874 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.545757    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
May 26 11:24:39.547162 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.547147    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-n4h7j"
May 26 11:24:39.548853 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.548827    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/91fba38c-504f-41ac-8b14-f8e6f9085f0b-etc-sysctl-conf\") pod \"tuned-qvkjq\" (UID: \"91fba38c-504f-41ac-8b14-f8e6f9085f0b\") " pod="openshift-cluster-node-tuning-operator/tuned-qvkjq"
May 26 11:24:39.548946 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.548862    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/91fba38c-504f-41ac-8b14-f8e6f9085f0b-lib-modules\") pod \"tuned-qvkjq\" (UID: \"91fba38c-504f-41ac-8b14-f8e6f9085f0b\") " pod="openshift-cluster-node-tuning-operator/tuned-qvkjq"
May 26 11:24:39.549304 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.549136    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/91fba38c-504f-41ac-8b14-f8e6f9085f0b-etc-tuned\") pod \"tuned-qvkjq\" (UID: \"91fba38c-504f-41ac-8b14-f8e6f9085f0b\") " pod="openshift-cluster-node-tuning-operator/tuned-qvkjq"
May 26 11:24:39.549304 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.549228    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3603f4d6-abda-490f-aa33-4c3f5eb284f2-iptables-alerter-script\") pod \"iptables-alerter-gzwdn\" (UID: \"3603f4d6-abda-490f-aa33-4c3f5eb284f2\") " pod="openshift-network-operator/iptables-alerter-gzwdn"
May 26 11:24:39.549304 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.549279    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/91fba38c-504f-41ac-8b14-f8e6f9085f0b-run\") pod \"tuned-qvkjq\" (UID: \"91fba38c-504f-41ac-8b14-f8e6f9085f0b\") " pod="openshift-cluster-node-tuning-operator/tuned-qvkjq"
May 26 11:24:39.549503 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.549318    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/91fba38c-504f-41ac-8b14-f8e6f9085f0b-sys\") pod \"tuned-qvkjq\" (UID: \"91fba38c-504f-41ac-8b14-f8e6f9085f0b\") " pod="openshift-cluster-node-tuning-operator/tuned-qvkjq"
May 26 11:24:39.549503 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.549352    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/91fba38c-504f-41ac-8b14-f8e6f9085f0b-tmp\") pod \"tuned-qvkjq\" (UID: \"91fba38c-504f-41ac-8b14-f8e6f9085f0b\") " pod="openshift-cluster-node-tuning-operator/tuned-qvkjq"
May 26 11:24:39.549503 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.549415    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kg48w\" (UniqueName: \"kubernetes.io/projected/240a9693-e4b9-485c-a138-6c3eb7f6734a-kube-api-access-kg48w\") pod \"node-ca-s9s2b\" (UID: \"240a9693-e4b9-485c-a138-6c3eb7f6734a\") " pod="openshift-image-registry/node-ca-s9s2b"
May 26 11:24:39.549503 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.549468    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bfb9\" (UniqueName: \"kubernetes.io/projected/3603f4d6-abda-490f-aa33-4c3f5eb284f2-kube-api-access-7bfb9\") pod \"iptables-alerter-gzwdn\" (UID: \"3603f4d6-abda-490f-aa33-4c3f5eb284f2\") " pod="openshift-network-operator/iptables-alerter-gzwdn"
May 26 11:24:39.549697 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.549506    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/91fba38c-504f-41ac-8b14-f8e6f9085f0b-etc-modprobe-d\") pod \"tuned-qvkjq\" (UID: \"91fba38c-504f-41ac-8b14-f8e6f9085f0b\") " pod="openshift-cluster-node-tuning-operator/tuned-qvkjq"
May 26 11:24:39.549697 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.549538    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/91fba38c-504f-41ac-8b14-f8e6f9085f0b-etc-systemd\") pod \"tuned-qvkjq\" (UID: \"91fba38c-504f-41ac-8b14-f8e6f9085f0b\") " pod="openshift-cluster-node-tuning-operator/tuned-qvkjq"
May 26 11:24:39.549697 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.549584    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/91fba38c-504f-41ac-8b14-f8e6f9085f0b-var-lib-kubelet\") pod \"tuned-qvkjq\" (UID: \"91fba38c-504f-41ac-8b14-f8e6f9085f0b\") " pod="openshift-cluster-node-tuning-operator/tuned-qvkjq"
May 26 11:24:39.549697 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.549647    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/91fba38c-504f-41ac-8b14-f8e6f9085f0b-host\") pod \"tuned-qvkjq\" (UID: \"91fba38c-504f-41ac-8b14-f8e6f9085f0b\") " pod="openshift-cluster-node-tuning-operator/tuned-qvkjq"
May 26 11:24:39.549697 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.549688    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/024ce274-4936-49fa-b084-ec86c46d34ed-tmp-dir\") pod \"node-resolver-4clzr\" (UID: \"024ce274-4936-49fa-b084-ec86c46d34ed\") " pod="openshift-dns/node-resolver-4clzr"
May 26 11:24:39.549936 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.549728    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lc9m\" (UniqueName: \"kubernetes.io/projected/024ce274-4936-49fa-b084-ec86c46d34ed-kube-api-access-7lc9m\") pod \"node-resolver-4clzr\" (UID: \"024ce274-4936-49fa-b084-ec86c46d34ed\") " pod="openshift-dns/node-resolver-4clzr"
May 26 11:24:39.549936 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.549765    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/240a9693-e4b9-485c-a138-6c3eb7f6734a-host\") pod \"node-ca-s9s2b\" (UID: \"240a9693-e4b9-485c-a138-6c3eb7f6734a\") " pod="openshift-image-registry/node-ca-s9s2b"
May 26 11:24:39.549936 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.549798    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/240a9693-e4b9-485c-a138-6c3eb7f6734a-serviceca\") pod \"node-ca-s9s2b\" (UID: \"240a9693-e4b9-485c-a138-6c3eb7f6734a\") " pod="openshift-image-registry/node-ca-s9s2b"
May 26 11:24:39.549936 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.549838    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3603f4d6-abda-490f-aa33-4c3f5eb284f2-host-slash\") pod \"iptables-alerter-gzwdn\" (UID: \"3603f4d6-abda-490f-aa33-4c3f5eb284f2\") " pod="openshift-network-operator/iptables-alerter-gzwdn"
May 26 11:24:39.549936 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.549880    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/91fba38c-504f-41ac-8b14-f8e6f9085f0b-etc-sysctl-d\") pod \"tuned-qvkjq\" (UID: \"91fba38c-504f-41ac-8b14-f8e6f9085f0b\") " pod="openshift-cluster-node-tuning-operator/tuned-qvkjq"
May 26 11:24:39.549936 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.549927    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fpx6g\" (UniqueName: \"kubernetes.io/projected/91fba38c-504f-41ac-8b14-f8e6f9085f0b-kube-api-access-fpx6g\") pod \"tuned-qvkjq\" (UID: \"91fba38c-504f-41ac-8b14-f8e6f9085f0b\") " pod="openshift-cluster-node-tuning-operator/tuned-qvkjq"
May 26 11:24:39.550235 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.549968    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/024ce274-4936-49fa-b084-ec86c46d34ed-hosts-file\") pod \"node-resolver-4clzr\" (UID: \"024ce274-4936-49fa-b084-ec86c46d34ed\") " pod="openshift-dns/node-resolver-4clzr"
May 26 11:24:39.550235 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.550000    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/91fba38c-504f-41ac-8b14-f8e6f9085f0b-etc-sysconfig\") pod \"tuned-qvkjq\" (UID: \"91fba38c-504f-41ac-8b14-f8e6f9085f0b\") " pod="openshift-cluster-node-tuning-operator/tuned-qvkjq"
May 26 11:24:39.550235 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.550030    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/91fba38c-504f-41ac-8b14-f8e6f9085f0b-etc-kubernetes\") pod \"tuned-qvkjq\" (UID: \"91fba38c-504f-41ac-8b14-f8e6f9085f0b\") " pod="openshift-cluster-node-tuning-operator/tuned-qvkjq"
May 26 11:24:39.552728 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.552707    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6g66v"
May 26 11:24:39.552855 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.552835    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-fq6hb"
May 26 11:24:39.555347 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.555329    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-js8cc"
May 26 11:24:39.556340 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.556321    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
May 26 11:24:39.556424 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.556356    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-r65rh\""
May 26 11:24:39.556591 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.556572    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
May 26 11:24:39.558345 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.558326    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
May 26 11:24:39.558431 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.558332    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
May 26 11:24:39.560274 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.560255    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
May 26 11:24:39.560274 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.560272    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-vpvdv\""
May 26 11:24:39.560711 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.560693    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
May 26 11:24:39.560933 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.560917    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
May 26 11:24:39.562018 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.562003    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
May 26 11:24:39.562687 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.562621    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
May 26 11:24:39.562687 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.562654    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
May 26 11:24:39.562844 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.562745    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-bk85b\""
May 26 11:24:39.562844 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.562752    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
May 26 11:24:39.563127 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.563109    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
May 26 11:24:39.563509 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.563493    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-mfb4g\""
May 26 11:24:39.589486 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.589454    2572 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-05-25 11:19:38 +0000 UTC" deadline="2027-11-13 11:18:48.481993593 +0000 UTC"
May 26 11:24:39.589486 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.589485    2572 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12863h54m8.892511695s"
May 26 11:24:39.638962 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.638925    2572 desired_state_of_world_populator.go:158] "Finished populating initial desired state of world"
May 26 11:24:39.651202 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.651142    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3603f4d6-abda-490f-aa33-4c3f5eb284f2-host-slash\") pod \"iptables-alerter-gzwdn\" (UID: \"3603f4d6-abda-490f-aa33-4c3f5eb284f2\") " pod="openshift-network-operator/iptables-alerter-gzwdn"
May 26 11:24:39.651361 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.651238    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3603f4d6-abda-490f-aa33-4c3f5eb284f2-host-slash\") pod \"iptables-alerter-gzwdn\" (UID: \"3603f4d6-abda-490f-aa33-4c3f5eb284f2\") " pod="openshift-network-operator/iptables-alerter-gzwdn"
May 26 11:24:39.651361 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.651240    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9e35a325-e1c3-4e2f-9ae5-b4120b6f1b60-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vjbr5\" (UID: \"9e35a325-e1c3-4e2f-9ae5-b4120b6f1b60\") " pod="openshift-multus/multus-additional-cni-plugins-vjbr5"
May 26 11:24:39.651361 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.651303    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjc5p\" (UniqueName: \"kubernetes.io/projected/16cccd56-9a2f-44f1-9571-664d5cde0cd7-kube-api-access-jjc5p\") pod \"aws-ebs-csi-driver-node-6g66v\" (UID: \"16cccd56-9a2f-44f1-9571-664d5cde0cd7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6g66v"
May 26 11:24:39.651361 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.651332    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/fa96a484-9200-4281-a4e2-30c04f5f8b04-agent-certs\") pod \"konnectivity-agent-n4h7j\" (UID: \"fa96a484-9200-4281-a4e2-30c04f5f8b04\") " pod="kube-system/konnectivity-agent-n4h7j"
May 26 11:24:39.651361 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.651360    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/977e7725-0a14-4446-a572-75ba05e7b12e-cni-binary-copy\") pod \"multus-fq6hb\" (UID: \"977e7725-0a14-4446-a572-75ba05e7b12e\") " pod="openshift-multus/multus-fq6hb"
May 26 11:24:39.651607 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.651386    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/977e7725-0a14-4446-a572-75ba05e7b12e-multus-socket-dir-parent\") pod \"multus-fq6hb\" (UID: \"977e7725-0a14-4446-a572-75ba05e7b12e\") " pod="openshift-multus/multus-fq6hb"
May 26 11:24:39.651607 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.651409    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/977e7725-0a14-4446-a572-75ba05e7b12e-etc-kubernetes\") pod \"multus-fq6hb\" (UID: \"977e7725-0a14-4446-a572-75ba05e7b12e\") " pod="openshift-multus/multus-fq6hb"
May 26 11:24:39.651607 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.651432    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7264c907-f536-4719-905a-2bbda9e3c336-var-lib-openvswitch\") pod \"ovnkube-node-js8cc\" (UID: \"7264c907-f536-4719-905a-2bbda9e3c336\") " pod="openshift-ovn-kubernetes/ovnkube-node-js8cc"
May 26 11:24:39.651607 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.651479    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9e35a325-e1c3-4e2f-9ae5-b4120b6f1b60-system-cni-dir\") pod \"multus-additional-cni-plugins-vjbr5\" (UID: \"9e35a325-e1c3-4e2f-9ae5-b4120b6f1b60\") " pod="openshift-multus/multus-additional-cni-plugins-vjbr5"
May 26 11:24:39.651607 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.651541    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7264c907-f536-4719-905a-2bbda9e3c336-run-systemd\") pod \"ovnkube-node-js8cc\" (UID: \"7264c907-f536-4719-905a-2bbda9e3c336\") " pod="openshift-ovn-kubernetes/ovnkube-node-js8cc"
May 26 11:24:39.651607 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.651583    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/9e35a325-e1c3-4e2f-9ae5-b4120b6f1b60-os-release\") pod \"multus-additional-cni-plugins-vjbr5\" (UID: \"9e35a325-e1c3-4e2f-9ae5-b4120b6f1b60\") " pod="openshift-multus/multus-additional-cni-plugins-vjbr5"
May 26 11:24:39.651856 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.651614    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/91fba38c-504f-41ac-8b14-f8e6f9085f0b-etc-modprobe-d\") pod \"tuned-qvkjq\" (UID: \"91fba38c-504f-41ac-8b14-f8e6f9085f0b\") " pod="openshift-cluster-node-tuning-operator/tuned-qvkjq"
May 26 11:24:39.651856 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.651640    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/91fba38c-504f-41ac-8b14-f8e6f9085f0b-host\") pod \"tuned-qvkjq\" (UID: \"91fba38c-504f-41ac-8b14-f8e6f9085f0b\") " pod="openshift-cluster-node-tuning-operator/tuned-qvkjq"
May 26 11:24:39.651856 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.651664    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/977e7725-0a14-4446-a572-75ba05e7b12e-host-var-lib-kubelet\") pod \"multus-fq6hb\" (UID: \"977e7725-0a14-4446-a572-75ba05e7b12e\") " pod="openshift-multus/multus-fq6hb"
May 26 11:24:39.651856 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.651683    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/977e7725-0a14-4446-a572-75ba05e7b12e-hostroot\") pod \"multus-fq6hb\" (UID: \"977e7725-0a14-4446-a572-75ba05e7b12e\") " pod="openshift-multus/multus-fq6hb"
May 26 11:24:39.651856 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.651715    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/977e7725-0a14-4446-a572-75ba05e7b12e-host-run-multus-certs\") pod \"multus-fq6hb\" (UID: \"977e7725-0a14-4446-a572-75ba05e7b12e\") " pod="openshift-multus/multus-fq6hb"
May 26 11:24:39.651856 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.651739    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7264c907-f536-4719-905a-2bbda9e3c336-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-js8cc\" (UID: \"7264c907-f536-4719-905a-2bbda9e3c336\") " pod="openshift-ovn-kubernetes/ovnkube-node-js8cc"
May 26 11:24:39.651856 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.651759    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/91fba38c-504f-41ac-8b14-f8e6f9085f0b-host\") pod \"tuned-qvkjq\" (UID: \"91fba38c-504f-41ac-8b14-f8e6f9085f0b\") " pod="openshift-cluster-node-tuning-operator/tuned-qvkjq"
May 26 11:24:39.651856 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.651768    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/024ce274-4936-49fa-b084-ec86c46d34ed-tmp-dir\") pod \"node-resolver-4clzr\" (UID: \"024ce274-4936-49fa-b084-ec86c46d34ed\") " pod="openshift-dns/node-resolver-4clzr"
May 26 11:24:39.651856 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.651767    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/91fba38c-504f-41ac-8b14-f8e6f9085f0b-etc-modprobe-d\") pod \"tuned-qvkjq\" (UID: \"91fba38c-504f-41ac-8b14-f8e6f9085f0b\") " pod="openshift-cluster-node-tuning-operator/tuned-qvkjq"
May 26 11:24:39.652236 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.651881    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/91fba38c-504f-41ac-8b14-f8e6f9085f0b-etc-sysctl-d\") pod \"tuned-qvkjq\" (UID: \"91fba38c-504f-41ac-8b14-f8e6f9085f0b\") " pod="openshift-cluster-node-tuning-operator/tuned-qvkjq"
May 26 11:24:39.652236 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.651924    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/16cccd56-9a2f-44f1-9571-664d5cde0cd7-etc-selinux\") pod \"aws-ebs-csi-driver-node-6g66v\" (UID: \"16cccd56-9a2f-44f1-9571-664d5cde0cd7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6g66v"
May 26 11:24:39.652236 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.651955    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7264c907-f536-4719-905a-2bbda9e3c336-log-socket\") pod \"ovnkube-node-js8cc\" (UID: \"7264c907-f536-4719-905a-2bbda9e3c336\") " pod="openshift-ovn-kubernetes/ovnkube-node-js8cc"
May 26 11:24:39.652236 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.651981    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/91fba38c-504f-41ac-8b14-f8e6f9085f0b-run\") pod \"tuned-qvkjq\" (UID: \"91fba38c-504f-41ac-8b14-f8e6f9085f0b\") " pod="openshift-cluster-node-tuning-operator/tuned-qvkjq"
May 26 11:24:39.652236 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.652006    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/91fba38c-504f-41ac-8b14-f8e6f9085f0b-sys\") pod \"tuned-qvkjq\" (UID: \"91fba38c-504f-41ac-8b14-f8e6f9085f0b\") " pod="openshift-cluster-node-tuning-operator/tuned-qvkjq"
May 26 11:24:39.652236 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.652034    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/91fba38c-504f-41ac-8b14-f8e6f9085f0b-etc-sysctl-d\") pod \"tuned-qvkjq\" (UID: \"91fba38c-504f-41ac-8b14-f8e6f9085f0b\") " pod="openshift-cluster-node-tuning-operator/tuned-qvkjq"
May 26 11:24:39.652236 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.652047    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/91fba38c-504f-41ac-8b14-f8e6f9085f0b-sys\") pod \"tuned-qvkjq\" (UID: \"91fba38c-504f-41ac-8b14-f8e6f9085f0b\") " pod="openshift-cluster-node-tuning-operator/tuned-qvkjq"
May 26 11:24:39.652236 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.652039    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/91fba38c-504f-41ac-8b14-f8e6f9085f0b-run\") pod \"tuned-qvkjq\" (UID: \"91fba38c-504f-41ac-8b14-f8e6f9085f0b\") " pod="openshift-cluster-node-tuning-operator/tuned-qvkjq"
May 26 11:24:39.652236 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.652049    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/91fba38c-504f-41ac-8b14-f8e6f9085f0b-tmp\") pod \"tuned-qvkjq\" (UID: \"91fba38c-504f-41ac-8b14-f8e6f9085f0b\") " pod="openshift-cluster-node-tuning-operator/tuned-qvkjq"
May 26 11:24:39.652236 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.652112    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/977e7725-0a14-4446-a572-75ba05e7b12e-host-var-lib-cni-bin\") pod \"multus-fq6hb\" (UID: \"977e7725-0a14-4446-a572-75ba05e7b12e\") " pod="openshift-multus/multus-fq6hb"
May 26 11:24:39.652236 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.652075    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/024ce274-4936-49fa-b084-ec86c46d34ed-tmp-dir\") pod \"node-resolver-4clzr\" (UID: \"024ce274-4936-49fa-b084-ec86c46d34ed\") " pod="openshift-dns/node-resolver-4clzr"
May 26 11:24:39.652236 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.652141    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7264c907-f536-4719-905a-2bbda9e3c336-node-log\") pod \"ovnkube-node-js8cc\" (UID: \"7264c907-f536-4719-905a-2bbda9e3c336\") " pod="openshift-ovn-kubernetes/ovnkube-node-js8cc"
May 26 11:24:39.652236 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.652166    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7264c907-f536-4719-905a-2bbda9e3c336-host-run-ovn-kubernetes\") pod \"ovnkube-node-js8cc\" (UID: \"7264c907-f536-4719-905a-2bbda9e3c336\") " pod="openshift-ovn-kubernetes/ovnkube-node-js8cc"
May 26 11:24:39.652236 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.652210    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9e35a325-e1c3-4e2f-9ae5-b4120b6f1b60-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vjbr5\" (UID: \"9e35a325-e1c3-4e2f-9ae5-b4120b6f1b60\") " pod="openshift-multus/multus-additional-cni-plugins-vjbr5"
May 26 11:24:39.652236 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.652243    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69l4f\" (UniqueName: \"kubernetes.io/projected/dede2966-ea61-4174-b667-c6fb46152d91-kube-api-access-69l4f\") pod \"network-metrics-daemon-k7jgt\" (UID: \"dede2966-ea61-4174-b667-c6fb46152d91\") " pod="openshift-multus/network-metrics-daemon-k7jgt"
May 26 11:24:39.652917 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.652268    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gbx6\" (UniqueName: \"kubernetes.io/projected/e87b53a4-1dca-46b1-bbd4-e484919093c4-kube-api-access-7gbx6\") pod \"network-check-target-cc5xb\" (UID: \"e87b53a4-1dca-46b1-bbd4-e484919093c4\") " pod="openshift-network-diagnostics/network-check-target-cc5xb"
May 26 11:24:39.652917 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.652294    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/fa96a484-9200-4281-a4e2-30c04f5f8b04-konnectivity-ca\") pod \"konnectivity-agent-n4h7j\" (UID: \"fa96a484-9200-4281-a4e2-30c04f5f8b04\") " pod="kube-system/konnectivity-agent-n4h7j"
May 26 11:24:39.652917 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.652330    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/977e7725-0a14-4446-a572-75ba05e7b12e-host-run-netns\") pod \"multus-fq6hb\" (UID: \"977e7725-0a14-4446-a572-75ba05e7b12e\") " pod="openshift-multus/multus-fq6hb"
May 26 11:24:39.652917 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.652357    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/977e7725-0a14-4446-a572-75ba05e7b12e-multus-daemon-config\") pod \"multus-fq6hb\" (UID: \"977e7725-0a14-4446-a572-75ba05e7b12e\") " pod="openshift-multus/multus-fq6hb"
May 26 11:24:39.652917 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.652382    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7264c907-f536-4719-905a-2bbda9e3c336-ovnkube-config\") pod \"ovnkube-node-js8cc\" (UID: \"7264c907-f536-4719-905a-2bbda9e3c336\") " pod="openshift-ovn-kubernetes/ovnkube-node-js8cc"
May 26 11:24:39.652917 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.652428    2572 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
May 26 11:24:39.652917 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.652443    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkxzs\" (UniqueName: \"kubernetes.io/projected/7264c907-f536-4719-905a-2bbda9e3c336-kube-api-access-lkxzs\") pod \"ovnkube-node-js8cc\" (UID: \"7264c907-f536-4719-905a-2bbda9e3c336\") " pod="openshift-ovn-kubernetes/ovnkube-node-js8cc"
May 26 11:24:39.652917 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.652515    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-7lc9m\" (UniqueName: \"kubernetes.io/projected/024ce274-4936-49fa-b084-ec86c46d34ed-kube-api-access-7lc9m\") pod \"node-resolver-4clzr\" (UID: \"024ce274-4936-49fa-b084-ec86c46d34ed\") " pod="openshift-dns/node-resolver-4clzr"
May 26 11:24:39.652917 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.652545    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/240a9693-e4b9-485c-a138-6c3eb7f6734a-serviceca\") pod \"node-ca-s9s2b\" (UID: \"240a9693-e4b9-485c-a138-6c3eb7f6734a\") " pod="openshift-image-registry/node-ca-s9s2b"
May 26 11:24:39.652917 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.652570    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-fpx6g\" (UniqueName: \"kubernetes.io/projected/91fba38c-504f-41ac-8b14-f8e6f9085f0b-kube-api-access-fpx6g\") pod \"tuned-qvkjq\" (UID: \"91fba38c-504f-41ac-8b14-f8e6f9085f0b\") " pod="openshift-cluster-node-tuning-operator/tuned-qvkjq"
May 26 11:24:39.652917 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.652596    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/977e7725-0a14-4446-a572-75ba05e7b12e-system-cni-dir\") pod \"multus-fq6hb\" (UID: \"977e7725-0a14-4446-a572-75ba05e7b12e\") " pod="openshift-multus/multus-fq6hb"
May 26 11:24:39.652917 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.652623    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7264c907-f536-4719-905a-2bbda9e3c336-run-openvswitch\") pod \"ovnkube-node-js8cc\" (UID: \"7264c907-f536-4719-905a-2bbda9e3c336\") " pod="openshift-ovn-kubernetes/ovnkube-node-js8cc"
May 26 11:24:39.652917 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.652688    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9e35a325-e1c3-4e2f-9ae5-b4120b6f1b60-cni-binary-copy\") pod \"multus-additional-cni-plugins-vjbr5\" (UID: \"9e35a325-e1c3-4e2f-9ae5-b4120b6f1b60\") " pod="openshift-multus/multus-additional-cni-plugins-vjbr5"
May 26 11:24:39.652917 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.652807    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6kgdk\" (UniqueName: \"kubernetes.io/projected/9e35a325-e1c3-4e2f-9ae5-b4120b6f1b60-kube-api-access-6kgdk\") pod \"multus-additional-cni-plugins-vjbr5\" (UID: \"9e35a325-e1c3-4e2f-9ae5-b4120b6f1b60\") " pod="openshift-multus/multus-additional-cni-plugins-vjbr5"
May 26 11:24:39.652917 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.652841    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/024ce274-4936-49fa-b084-ec86c46d34ed-hosts-file\") pod \"node-resolver-4clzr\" (UID: \"024ce274-4936-49fa-b084-ec86c46d34ed\") " pod="openshift-dns/node-resolver-4clzr"
May 26 11:24:39.652917 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.652878    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/91fba38c-504f-41ac-8b14-f8e6f9085f0b-etc-sysconfig\") pod \"tuned-qvkjq\" (UID: \"91fba38c-504f-41ac-8b14-f8e6f9085f0b\") " pod="openshift-cluster-node-tuning-operator/tuned-qvkjq"
May 26 11:24:39.652917 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.652892    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/024ce274-4936-49fa-b084-ec86c46d34ed-hosts-file\") pod \"node-resolver-4clzr\" (UID: \"024ce274-4936-49fa-b084-ec86c46d34ed\") " pod="openshift-dns/node-resolver-4clzr"
May 26 11:24:39.653600 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.652907    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/91fba38c-504f-41ac-8b14-f8e6f9085f0b-etc-sysctl-conf\") pod \"tuned-qvkjq\" (UID: \"91fba38c-504f-41ac-8b14-f8e6f9085f0b\") " pod="openshift-cluster-node-tuning-operator/tuned-qvkjq"
May 26 11:24:39.653600 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.652932    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/91fba38c-504f-41ac-8b14-f8e6f9085f0b-lib-modules\") pod \"tuned-qvkjq\" (UID: \"91fba38c-504f-41ac-8b14-f8e6f9085f0b\") " pod="openshift-cluster-node-tuning-operator/tuned-qvkjq"
May 26 11:24:39.653600 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.652984    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/977e7725-0a14-4446-a572-75ba05e7b12e-cnibin\") pod \"multus-fq6hb\" (UID: \"977e7725-0a14-4446-a572-75ba05e7b12e\") " pod="openshift-multus/multus-fq6hb"
May 26 11:24:39.653600 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.652987    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/91fba38c-504f-41ac-8b14-f8e6f9085f0b-etc-sysconfig\") pod \"tuned-qvkjq\" (UID: \"91fba38c-504f-41ac-8b14-f8e6f9085f0b\") " pod="openshift-cluster-node-tuning-operator/tuned-qvkjq"
May 26 11:24:39.653600 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.653018    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/240a9693-e4b9-485c-a138-6c3eb7f6734a-serviceca\") pod \"node-ca-s9s2b\" (UID: \"240a9693-e4b9-485c-a138-6c3eb7f6734a\") " pod="openshift-image-registry/node-ca-s9s2b"
May 26 11:24:39.653600 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.653029    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/977e7725-0a14-4446-a572-75ba05e7b12e-multus-conf-dir\") pod \"multus-fq6hb\" (UID: \"977e7725-0a14-4446-a572-75ba05e7b12e\") " pod="openshift-multus/multus-fq6hb"
May 26 11:24:39.653600 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.653053    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gsjth\" (UniqueName: \"kubernetes.io/projected/977e7725-0a14-4446-a572-75ba05e7b12e-kube-api-access-gsjth\") pod \"multus-fq6hb\" (UID: \"977e7725-0a14-4446-a572-75ba05e7b12e\") " pod="openshift-multus/multus-fq6hb"
May 26 11:24:39.653600 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.653088    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/91fba38c-504f-41ac-8b14-f8e6f9085f0b-etc-sysctl-conf\") pod \"tuned-qvkjq\" (UID: \"91fba38c-504f-41ac-8b14-f8e6f9085f0b\") " pod="openshift-cluster-node-tuning-operator/tuned-qvkjq"
May 26 11:24:39.653600 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.653082    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7264c907-f536-4719-905a-2bbda9e3c336-systemd-units\") pod \"ovnkube-node-js8cc\" (UID: \"7264c907-f536-4719-905a-2bbda9e3c336\") " pod="openshift-ovn-kubernetes/ovnkube-node-js8cc"
May 26 11:24:39.653600 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.653128    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3603f4d6-abda-490f-aa33-4c3f5eb284f2-iptables-alerter-script\") pod \"iptables-alerter-gzwdn\" (UID: \"3603f4d6-abda-490f-aa33-4c3f5eb284f2\") " pod="openshift-network-operator/iptables-alerter-gzwdn"
May 26 11:24:39.653600 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.653095    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/91fba38c-504f-41ac-8b14-f8e6f9085f0b-lib-modules\") pod \"tuned-qvkjq\" (UID: \"91fba38c-504f-41ac-8b14-f8e6f9085f0b\") " pod="openshift-cluster-node-tuning-operator/tuned-qvkjq"
May 26 11:24:39.653600 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.653156    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/16cccd56-9a2f-44f1-9571-664d5cde0cd7-registration-dir\") pod \"aws-ebs-csi-driver-node-6g66v\" (UID: \"16cccd56-9a2f-44f1-9571-664d5cde0cd7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6g66v"
May 26 11:24:39.653600 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.653216    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7264c907-f536-4719-905a-2bbda9e3c336-host-kubelet\") pod \"ovnkube-node-js8cc\" (UID: \"7264c907-f536-4719-905a-2bbda9e3c336\") " pod="openshift-ovn-kubernetes/ovnkube-node-js8cc"
May 26 11:24:39.653600 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.653246    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7264c907-f536-4719-905a-2bbda9e3c336-etc-openvswitch\") pod \"ovnkube-node-js8cc\" (UID: \"7264c907-f536-4719-905a-2bbda9e3c336\") " pod="openshift-ovn-kubernetes/ovnkube-node-js8cc"
May 26 11:24:39.653600 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.653272    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-7bfb9\" (UniqueName: \"kubernetes.io/projected/3603f4d6-abda-490f-aa33-4c3f5eb284f2-kube-api-access-7bfb9\") pod \"iptables-alerter-gzwdn\" (UID: \"3603f4d6-abda-490f-aa33-4c3f5eb284f2\") " pod="openshift-network-operator/iptables-alerter-gzwdn"
May 26 11:24:39.653600 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.653302    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/91fba38c-504f-41ac-8b14-f8e6f9085f0b-etc-systemd\") pod \"tuned-qvkjq\" (UID: \"91fba38c-504f-41ac-8b14-f8e6f9085f0b\") " pod="openshift-cluster-node-tuning-operator/tuned-qvkjq"
May 26 11:24:39.653600 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.653326    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/16cccd56-9a2f-44f1-9571-664d5cde0cd7-device-dir\") pod \"aws-ebs-csi-driver-node-6g66v\" (UID: \"16cccd56-9a2f-44f1-9571-664d5cde0cd7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6g66v"
May 26 11:24:39.654305 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.653350    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/16cccd56-9a2f-44f1-9571-664d5cde0cd7-sys-fs\") pod \"aws-ebs-csi-driver-node-6g66v\" (UID: \"16cccd56-9a2f-44f1-9571-664d5cde0cd7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6g66v"
May 26 11:24:39.654305 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.653400    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/91fba38c-504f-41ac-8b14-f8e6f9085f0b-etc-systemd\") pod \"tuned-qvkjq\" (UID: \"91fba38c-504f-41ac-8b14-f8e6f9085f0b\") " pod="openshift-cluster-node-tuning-operator/tuned-qvkjq"
May 26 11:24:39.654305 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.653437    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dede2966-ea61-4174-b667-c6fb46152d91-metrics-certs\") pod \"network-metrics-daemon-k7jgt\" (UID: \"dede2966-ea61-4174-b667-c6fb46152d91\") " pod="openshift-multus/network-metrics-daemon-k7jgt"
May 26 11:24:39.654305 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.653465    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7264c907-f536-4719-905a-2bbda9e3c336-host-run-netns\") pod \"ovnkube-node-js8cc\" (UID: \"7264c907-f536-4719-905a-2bbda9e3c336\") " pod="openshift-ovn-kubernetes/ovnkube-node-js8cc"
May 26 11:24:39.654305 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.653491    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/977e7725-0a14-4446-a572-75ba05e7b12e-host-run-k8s-cni-cncf-io\") pod \"multus-fq6hb\" (UID: \"977e7725-0a14-4446-a572-75ba05e7b12e\") " pod="openshift-multus/multus-fq6hb"
May 26 11:24:39.654305 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.653516    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7264c907-f536-4719-905a-2bbda9e3c336-host-cni-netd\") pod \"ovnkube-node-js8cc\" (UID: \"7264c907-f536-4719-905a-2bbda9e3c336\") " pod="openshift-ovn-kubernetes/ovnkube-node-js8cc"
May 26 11:24:39.654305 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.653543    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7264c907-f536-4719-905a-2bbda9e3c336-ovn-node-metrics-cert\") pod \"ovnkube-node-js8cc\" (UID: \"7264c907-f536-4719-905a-2bbda9e3c336\") " pod="openshift-ovn-kubernetes/ovnkube-node-js8cc"
May 26 11:24:39.654305 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.653569    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/91fba38c-504f-41ac-8b14-f8e6f9085f0b-etc-kubernetes\") pod \"tuned-qvkjq\" (UID: \"91fba38c-504f-41ac-8b14-f8e6f9085f0b\") " pod="openshift-cluster-node-tuning-operator/tuned-qvkjq"
May 26 11:24:39.654305 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.653599    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/91fba38c-504f-41ac-8b14-f8e6f9085f0b-etc-tuned\") pod \"tuned-qvkjq\" (UID: \"91fba38c-504f-41ac-8b14-f8e6f9085f0b\") " pod="openshift-cluster-node-tuning-operator/tuned-qvkjq"
May 26 11:24:39.654305 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.653624    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/977e7725-0a14-4446-a572-75ba05e7b12e-os-release\") pod \"multus-fq6hb\" (UID: \"977e7725-0a14-4446-a572-75ba05e7b12e\") " pod="openshift-multus/multus-fq6hb"
May 26 11:24:39.654305 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.653639    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3603f4d6-abda-490f-aa33-4c3f5eb284f2-iptables-alerter-script\") pod \"iptables-alerter-gzwdn\" (UID: \"3603f4d6-abda-490f-aa33-4c3f5eb284f2\") " pod="openshift-network-operator/iptables-alerter-gzwdn"
May 26 11:24:39.654305 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.653648    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/977e7725-0a14-4446-a572-75ba05e7b12e-host-var-lib-cni-multus\") pod \"multus-fq6hb\" (UID: \"977e7725-0a14-4446-a572-75ba05e7b12e\") " pod="openshift-multus/multus-fq6hb"
May 26 11:24:39.654305 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.653648    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/91fba38c-504f-41ac-8b14-f8e6f9085f0b-etc-kubernetes\") pod \"tuned-qvkjq\" (UID: \"91fba38c-504f-41ac-8b14-f8e6f9085f0b\") " pod="openshift-cluster-node-tuning-operator/tuned-qvkjq"
May 26 11:24:39.654305 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.653673    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7264c907-f536-4719-905a-2bbda9e3c336-host-slash\") pod \"ovnkube-node-js8cc\" (UID: \"7264c907-f536-4719-905a-2bbda9e3c336\") " pod="openshift-ovn-kubernetes/ovnkube-node-js8cc"
May 26 11:24:39.654305 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.653698    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7264c907-f536-4719-905a-2bbda9e3c336-env-overrides\") pod \"ovnkube-node-js8cc\" (UID: \"7264c907-f536-4719-905a-2bbda9e3c336\") " pod="openshift-ovn-kubernetes/ovnkube-node-js8cc"
May 26 11:24:39.654305 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.653719    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9e35a325-e1c3-4e2f-9ae5-b4120b6f1b60-cnibin\") pod \"multus-additional-cni-plugins-vjbr5\" (UID: \"9e35a325-e1c3-4e2f-9ae5-b4120b6f1b60\") " pod="openshift-multus/multus-additional-cni-plugins-vjbr5"
May 26 11:24:39.654305 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.653743    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9e35a325-e1c3-4e2f-9ae5-b4120b6f1b60-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vjbr5\" (UID: \"9e35a325-e1c3-4e2f-9ae5-b4120b6f1b60\") " pod="openshift-multus/multus-additional-cni-plugins-vjbr5"
May 26 11:24:39.654779 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.653767    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/16cccd56-9a2f-44f1-9571-664d5cde0cd7-socket-dir\") pod \"aws-ebs-csi-driver-node-6g66v\" (UID: \"16cccd56-9a2f-44f1-9571-664d5cde0cd7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6g66v"
May 26 11:24:39.654779 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.653801    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-kg48w\" (UniqueName: \"kubernetes.io/projected/240a9693-e4b9-485c-a138-6c3eb7f6734a-kube-api-access-kg48w\") pod \"node-ca-s9s2b\" (UID: \"240a9693-e4b9-485c-a138-6c3eb7f6734a\") " pod="openshift-image-registry/node-ca-s9s2b"
May 26 11:24:39.654779 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.653854    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/91fba38c-504f-41ac-8b14-f8e6f9085f0b-var-lib-kubelet\") pod \"tuned-qvkjq\" (UID: \"91fba38c-504f-41ac-8b14-f8e6f9085f0b\") " pod="openshift-cluster-node-tuning-operator/tuned-qvkjq"
May 26 11:24:39.654779 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.653925    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/16cccd56-9a2f-44f1-9571-664d5cde0cd7-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6g66v\" (UID: \"16cccd56-9a2f-44f1-9571-664d5cde0cd7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6g66v"
May 26 11:24:39.654779 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.653940    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/91fba38c-504f-41ac-8b14-f8e6f9085f0b-var-lib-kubelet\") pod \"tuned-qvkjq\" (UID: \"91fba38c-504f-41ac-8b14-f8e6f9085f0b\") " pod="openshift-cluster-node-tuning-operator/tuned-qvkjq"
May 26 11:24:39.654779 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.653978    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/977e7725-0a14-4446-a572-75ba05e7b12e-multus-cni-dir\") pod \"multus-fq6hb\" (UID: \"977e7725-0a14-4446-a572-75ba05e7b12e\") " pod="openshift-multus/multus-fq6hb"
May 26 11:24:39.654779 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.654021    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7264c907-f536-4719-905a-2bbda9e3c336-run-ovn\") pod \"ovnkube-node-js8cc\" (UID: \"7264c907-f536-4719-905a-2bbda9e3c336\") " pod="openshift-ovn-kubernetes/ovnkube-node-js8cc"
May 26 11:24:39.654779 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.654056    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7264c907-f536-4719-905a-2bbda9e3c336-host-cni-bin\") pod \"ovnkube-node-js8cc\" (UID: \"7264c907-f536-4719-905a-2bbda9e3c336\") " pod="openshift-ovn-kubernetes/ovnkube-node-js8cc"
May 26 11:24:39.654779 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.654080    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7264c907-f536-4719-905a-2bbda9e3c336-ovnkube-script-lib\") pod \"ovnkube-node-js8cc\" (UID: \"7264c907-f536-4719-905a-2bbda9e3c336\") " pod="openshift-ovn-kubernetes/ovnkube-node-js8cc"
May 26 11:24:39.654779 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.654109    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/240a9693-e4b9-485c-a138-6c3eb7f6734a-host\") pod \"node-ca-s9s2b\" (UID: \"240a9693-e4b9-485c-a138-6c3eb7f6734a\") " pod="openshift-image-registry/node-ca-s9s2b"
May 26 11:24:39.654779 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.654230    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/240a9693-e4b9-485c-a138-6c3eb7f6734a-host\") pod \"node-ca-s9s2b\" (UID: \"240a9693-e4b9-485c-a138-6c3eb7f6734a\") " pod="openshift-image-registry/node-ca-s9s2b"
May 26 11:24:39.655670 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.655649    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/91fba38c-504f-41ac-8b14-f8e6f9085f0b-tmp\") pod \"tuned-qvkjq\" (UID: \"91fba38c-504f-41ac-8b14-f8e6f9085f0b\") " pod="openshift-cluster-node-tuning-operator/tuned-qvkjq"
May 26 11:24:39.656059 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.656028    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/91fba38c-504f-41ac-8b14-f8e6f9085f0b-etc-tuned\") pod \"tuned-qvkjq\" (UID: \"91fba38c-504f-41ac-8b14-f8e6f9085f0b\") " pod="openshift-cluster-node-tuning-operator/tuned-qvkjq"
May 26 11:24:39.674554 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.674530    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lc9m\" (UniqueName: \"kubernetes.io/projected/024ce274-4936-49fa-b084-ec86c46d34ed-kube-api-access-7lc9m\") pod \"node-resolver-4clzr\" (UID: \"024ce274-4936-49fa-b084-ec86c46d34ed\") " pod="openshift-dns/node-resolver-4clzr"
May 26 11:24:39.674722 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.674700    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kg48w\" (UniqueName: \"kubernetes.io/projected/240a9693-e4b9-485c-a138-6c3eb7f6734a-kube-api-access-kg48w\") pod \"node-ca-s9s2b\" (UID: \"240a9693-e4b9-485c-a138-6c3eb7f6734a\") " pod="openshift-image-registry/node-ca-s9s2b"
May 26 11:24:39.675180 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.675160    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpx6g\" (UniqueName: \"kubernetes.io/projected/91fba38c-504f-41ac-8b14-f8e6f9085f0b-kube-api-access-fpx6g\") pod \"tuned-qvkjq\" (UID: \"91fba38c-504f-41ac-8b14-f8e6f9085f0b\") " pod="openshift-cluster-node-tuning-operator/tuned-qvkjq"
May 26 11:24:39.675678 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.675650    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bfb9\" (UniqueName: \"kubernetes.io/projected/3603f4d6-abda-490f-aa33-4c3f5eb284f2-kube-api-access-7bfb9\") pod \"iptables-alerter-gzwdn\" (UID: \"3603f4d6-abda-490f-aa33-4c3f5eb284f2\") " pod="openshift-network-operator/iptables-alerter-gzwdn"
May 26 11:24:39.754648 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.754608    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dede2966-ea61-4174-b667-c6fb46152d91-metrics-certs\") pod \"network-metrics-daemon-k7jgt\" (UID: \"dede2966-ea61-4174-b667-c6fb46152d91\") " pod="openshift-multus/network-metrics-daemon-k7jgt"
May 26 11:24:39.754648 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.754656    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7264c907-f536-4719-905a-2bbda9e3c336-host-run-netns\") pod \"ovnkube-node-js8cc\" (UID: \"7264c907-f536-4719-905a-2bbda9e3c336\") " pod="openshift-ovn-kubernetes/ovnkube-node-js8cc"
May 26 11:24:39.754888 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.754683    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/977e7725-0a14-4446-a572-75ba05e7b12e-host-run-k8s-cni-cncf-io\") pod \"multus-fq6hb\" (UID: \"977e7725-0a14-4446-a572-75ba05e7b12e\") " pod="openshift-multus/multus-fq6hb"
May 26 11:24:39.754888 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.754705    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7264c907-f536-4719-905a-2bbda9e3c336-host-cni-netd\") pod \"ovnkube-node-js8cc\" (UID: \"7264c907-f536-4719-905a-2bbda9e3c336\") " pod="openshift-ovn-kubernetes/ovnkube-node-js8cc"
May 26 11:24:39.754888 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.754724    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7264c907-f536-4719-905a-2bbda9e3c336-ovn-node-metrics-cert\") pod \"ovnkube-node-js8cc\" (UID: \"7264c907-f536-4719-905a-2bbda9e3c336\") " pod="openshift-ovn-kubernetes/ovnkube-node-js8cc"
May 26 11:24:39.754888 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.754747    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/977e7725-0a14-4446-a572-75ba05e7b12e-os-release\") pod \"multus-fq6hb\" (UID: \"977e7725-0a14-4446-a572-75ba05e7b12e\") " pod="openshift-multus/multus-fq6hb"
May 26 11:24:39.754888 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.754768    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/977e7725-0a14-4446-a572-75ba05e7b12e-host-var-lib-cni-multus\") pod \"multus-fq6hb\" (UID: \"977e7725-0a14-4446-a572-75ba05e7b12e\") " pod="openshift-multus/multus-fq6hb"
May 26 11:24:39.754888 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.754790    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7264c907-f536-4719-905a-2bbda9e3c336-host-slash\") pod \"ovnkube-node-js8cc\" (UID: \"7264c907-f536-4719-905a-2bbda9e3c336\") " pod="openshift-ovn-kubernetes/ovnkube-node-js8cc"
May 26 11:24:39.754888 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.754810    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7264c907-f536-4719-905a-2bbda9e3c336-env-overrides\") pod \"ovnkube-node-js8cc\" (UID: \"7264c907-f536-4719-905a-2bbda9e3c336\") " pod="openshift-ovn-kubernetes/ovnkube-node-js8cc"
May 26 11:24:39.754888 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.754833    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9e35a325-e1c3-4e2f-9ae5-b4120b6f1b60-cnibin\") pod \"multus-additional-cni-plugins-vjbr5\" (UID: \"9e35a325-e1c3-4e2f-9ae5-b4120b6f1b60\") " pod="openshift-multus/multus-additional-cni-plugins-vjbr5"
May 26 11:24:39.754888 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.754859    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9e35a325-e1c3-4e2f-9ae5-b4120b6f1b60-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vjbr5\" (UID: \"9e35a325-e1c3-4e2f-9ae5-b4120b6f1b60\") " pod="openshift-multus/multus-additional-cni-plugins-vjbr5"
May 26 11:24:39.755310 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.754884    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/16cccd56-9a2f-44f1-9571-664d5cde0cd7-socket-dir\") pod \"aws-ebs-csi-driver-node-6g66v\" (UID: \"16cccd56-9a2f-44f1-9571-664d5cde0cd7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6g66v"
May 26 11:24:39.755310 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.754919    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/16cccd56-9a2f-44f1-9571-664d5cde0cd7-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6g66v\" (UID: \"16cccd56-9a2f-44f1-9571-664d5cde0cd7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6g66v"
May 26 11:24:39.755310 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.754943    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/977e7725-0a14-4446-a572-75ba05e7b12e-multus-cni-dir\") pod \"multus-fq6hb\" (UID: \"977e7725-0a14-4446-a572-75ba05e7b12e\") " pod="openshift-multus/multus-fq6hb"
May 26 11:24:39.755310 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.754967    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7264c907-f536-4719-905a-2bbda9e3c336-run-ovn\") pod \"ovnkube-node-js8cc\" (UID: \"7264c907-f536-4719-905a-2bbda9e3c336\") " pod="openshift-ovn-kubernetes/ovnkube-node-js8cc"
May 26 11:24:39.755310 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.754991    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7264c907-f536-4719-905a-2bbda9e3c336-host-cni-bin\") pod \"ovnkube-node-js8cc\" (UID: \"7264c907-f536-4719-905a-2bbda9e3c336\") " pod="openshift-ovn-kubernetes/ovnkube-node-js8cc"
May 26 11:24:39.755310 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.755017    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7264c907-f536-4719-905a-2bbda9e3c336-ovnkube-script-lib\") pod \"ovnkube-node-js8cc\" (UID: \"7264c907-f536-4719-905a-2bbda9e3c336\") " pod="openshift-ovn-kubernetes/ovnkube-node-js8cc"
May 26 11:24:39.755310 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.755048    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9e35a325-e1c3-4e2f-9ae5-b4120b6f1b60-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vjbr5\" (UID: \"9e35a325-e1c3-4e2f-9ae5-b4120b6f1b60\") " pod="openshift-multus/multus-additional-cni-plugins-vjbr5"
May 26 11:24:39.755310 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.755066    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7264c907-f536-4719-905a-2bbda9e3c336-host-slash\") pod \"ovnkube-node-js8cc\" (UID: \"7264c907-f536-4719-905a-2bbda9e3c336\") " pod="openshift-ovn-kubernetes/ovnkube-node-js8cc"
May 26 11:24:39.755310 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.755077    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-jjc5p\" (UniqueName: \"kubernetes.io/projected/16cccd56-9a2f-44f1-9571-664d5cde0cd7-kube-api-access-jjc5p\") pod \"aws-ebs-csi-driver-node-6g66v\" (UID: \"16cccd56-9a2f-44f1-9571-664d5cde0cd7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6g66v"
May 26 11:24:39.755310 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.755105    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/fa96a484-9200-4281-a4e2-30c04f5f8b04-agent-certs\") pod \"konnectivity-agent-n4h7j\" (UID: \"fa96a484-9200-4281-a4e2-30c04f5f8b04\") " pod="kube-system/konnectivity-agent-n4h7j"
May 26 11:24:39.755738 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.755535    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9e35a325-e1c3-4e2f-9ae5-b4120b6f1b60-cnibin\") pod \"multus-additional-cni-plugins-vjbr5\" (UID: \"9e35a325-e1c3-4e2f-9ae5-b4120b6f1b60\") " pod="openshift-multus/multus-additional-cni-plugins-vjbr5"
May 26 11:24:39.755738 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.755524    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9e35a325-e1c3-4e2f-9ae5-b4120b6f1b60-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vjbr5\" (UID: \"9e35a325-e1c3-4e2f-9ae5-b4120b6f1b60\") " pod="openshift-multus/multus-additional-cni-plugins-vjbr5"
May 26 11:24:39.755738 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.755600    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/977e7725-0a14-4446-a572-75ba05e7b12e-host-run-k8s-cni-cncf-io\") pod \"multus-fq6hb\" (UID: \"977e7725-0a14-4446-a572-75ba05e7b12e\") " pod="openshift-multus/multus-fq6hb"
May 26 11:24:39.755738 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.755601    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/977e7725-0a14-4446-a572-75ba05e7b12e-host-var-lib-cni-multus\") pod \"multus-fq6hb\" (UID: \"977e7725-0a14-4446-a572-75ba05e7b12e\") " pod="openshift-multus/multus-fq6hb"
May 26 11:24:39.755738 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.755618    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/16cccd56-9a2f-44f1-9571-664d5cde0cd7-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6g66v\" (UID: \"16cccd56-9a2f-44f1-9571-664d5cde0cd7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6g66v"
May 26 11:24:39.755738 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.755632    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7264c907-f536-4719-905a-2bbda9e3c336-host-cni-bin\") pod \"ovnkube-node-js8cc\" (UID: \"7264c907-f536-4719-905a-2bbda9e3c336\") " pod="openshift-ovn-kubernetes/ovnkube-node-js8cc"
May 26 11:24:39.755738 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.755669    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7264c907-f536-4719-905a-2bbda9e3c336-host-cni-netd\") pod \"ovnkube-node-js8cc\" (UID: \"7264c907-f536-4719-905a-2bbda9e3c336\") " pod="openshift-ovn-kubernetes/ovnkube-node-js8cc"
May 26 11:24:39.755738 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:24:39.755707    2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 26 11:24:39.755738 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.755716    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7264c907-f536-4719-905a-2bbda9e3c336-host-run-netns\") pod \"ovnkube-node-js8cc\" (UID: \"7264c907-f536-4719-905a-2bbda9e3c336\") " pod="openshift-ovn-kubernetes/ovnkube-node-js8cc"
May 26 11:24:39.756132 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.755746    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/16cccd56-9a2f-44f1-9571-664d5cde0cd7-socket-dir\") pod \"aws-ebs-csi-driver-node-6g66v\" (UID: \"16cccd56-9a2f-44f1-9571-664d5cde0cd7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6g66v"
May 26 11:24:39.756132 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.755758    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7264c907-f536-4719-905a-2bbda9e3c336-run-ovn\") pod \"ovnkube-node-js8cc\" (UID: \"7264c907-f536-4719-905a-2bbda9e3c336\") " pod="openshift-ovn-kubernetes/ovnkube-node-js8cc"
May 26 11:24:39.756132 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:24:39.755897    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dede2966-ea61-4174-b667-c6fb46152d91-metrics-certs podName:dede2966-ea61-4174-b667-c6fb46152d91 nodeName:}" failed. No retries permitted until 2026-05-26 11:24:40.255850298 +0000 UTC m=+3.158060610 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dede2966-ea61-4174-b667-c6fb46152d91-metrics-certs") pod "network-metrics-daemon-k7jgt" (UID: "dede2966-ea61-4174-b667-c6fb46152d91") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 26 11:24:39.756132 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.755896    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7264c907-f536-4719-905a-2bbda9e3c336-env-overrides\") pod \"ovnkube-node-js8cc\" (UID: \"7264c907-f536-4719-905a-2bbda9e3c336\") " pod="openshift-ovn-kubernetes/ovnkube-node-js8cc"
May 26 11:24:39.756132 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.755973    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/977e7725-0a14-4446-a572-75ba05e7b12e-os-release\") pod \"multus-fq6hb\" (UID: \"977e7725-0a14-4446-a572-75ba05e7b12e\") " pod="openshift-multus/multus-fq6hb"
May 26 11:24:39.756132 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.756012    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9e35a325-e1c3-4e2f-9ae5-b4120b6f1b60-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vjbr5\" (UID: \"9e35a325-e1c3-4e2f-9ae5-b4120b6f1b60\") " pod="openshift-multus/multus-additional-cni-plugins-vjbr5"
May 26 11:24:39.756438 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.756148    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/977e7725-0a14-4446-a572-75ba05e7b12e-multus-cni-dir\") pod \"multus-fq6hb\" (UID: \"977e7725-0a14-4446-a572-75ba05e7b12e\") " pod="openshift-multus/multus-fq6hb"
May 26 11:24:39.756438 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.756221    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/977e7725-0a14-4446-a572-75ba05e7b12e-cni-binary-copy\") pod \"multus-fq6hb\" (UID: \"977e7725-0a14-4446-a572-75ba05e7b12e\") " pod="openshift-multus/multus-fq6hb"
May 26 11:24:39.756438 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.756275    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/977e7725-0a14-4446-a572-75ba05e7b12e-multus-socket-dir-parent\") pod \"multus-fq6hb\" (UID: \"977e7725-0a14-4446-a572-75ba05e7b12e\") " pod="openshift-multus/multus-fq6hb"
May 26 11:24:39.756438 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.756304    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/977e7725-0a14-4446-a572-75ba05e7b12e-etc-kubernetes\") pod \"multus-fq6hb\" (UID: \"977e7725-0a14-4446-a572-75ba05e7b12e\") " pod="openshift-multus/multus-fq6hb"
May 26 11:24:39.756438 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.756306    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7264c907-f536-4719-905a-2bbda9e3c336-ovnkube-script-lib\") pod \"ovnkube-node-js8cc\" (UID: \"7264c907-f536-4719-905a-2bbda9e3c336\") " pod="openshift-ovn-kubernetes/ovnkube-node-js8cc"
May 26 11:24:39.756438 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.756342    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7264c907-f536-4719-905a-2bbda9e3c336-var-lib-openvswitch\") pod \"ovnkube-node-js8cc\" (UID: \"7264c907-f536-4719-905a-2bbda9e3c336\") " pod="openshift-ovn-kubernetes/ovnkube-node-js8cc"
May 26 11:24:39.756438 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.756368    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9e35a325-e1c3-4e2f-9ae5-b4120b6f1b60-system-cni-dir\") pod \"multus-additional-cni-plugins-vjbr5\" (UID: \"9e35a325-e1c3-4e2f-9ae5-b4120b6f1b60\") " pod="openshift-multus/multus-additional-cni-plugins-vjbr5"
May 26 11:24:39.756438 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.756393    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7264c907-f536-4719-905a-2bbda9e3c336-run-systemd\") pod \"ovnkube-node-js8cc\" (UID: \"7264c907-f536-4719-905a-2bbda9e3c336\") " pod="openshift-ovn-kubernetes/ovnkube-node-js8cc"
May 26 11:24:39.756438 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.756431    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/9e35a325-e1c3-4e2f-9ae5-b4120b6f1b60-os-release\") pod \"multus-additional-cni-plugins-vjbr5\" (UID: \"9e35a325-e1c3-4e2f-9ae5-b4120b6f1b60\") " pod="openshift-multus/multus-additional-cni-plugins-vjbr5"
May 26 11:24:39.756855 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.756458    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/977e7725-0a14-4446-a572-75ba05e7b12e-host-var-lib-kubelet\") pod \"multus-fq6hb\" (UID: \"977e7725-0a14-4446-a572-75ba05e7b12e\") " pod="openshift-multus/multus-fq6hb"
May 26 11:24:39.756855 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.756484    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/977e7725-0a14-4446-a572-75ba05e7b12e-hostroot\") pod \"multus-fq6hb\" (UID: \"977e7725-0a14-4446-a572-75ba05e7b12e\") " pod="openshift-multus/multus-fq6hb"
May 26 11:24:39.756855 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.756510    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/977e7725-0a14-4446-a572-75ba05e7b12e-host-run-multus-certs\") pod \"multus-fq6hb\" (UID: \"977e7725-0a14-4446-a572-75ba05e7b12e\") " pod="openshift-multus/multus-fq6hb"
May 26 11:24:39.756855 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.756554    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7264c907-f536-4719-905a-2bbda9e3c336-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-js8cc\" (UID: \"7264c907-f536-4719-905a-2bbda9e3c336\") " pod="openshift-ovn-kubernetes/ovnkube-node-js8cc"
May 26 11:24:39.756855 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.756586    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/16cccd56-9a2f-44f1-9571-664d5cde0cd7-etc-selinux\") pod \"aws-ebs-csi-driver-node-6g66v\" (UID: \"16cccd56-9a2f-44f1-9571-664d5cde0cd7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6g66v"
May 26 11:24:39.756855 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.756611    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7264c907-f536-4719-905a-2bbda9e3c336-log-socket\") pod \"ovnkube-node-js8cc\" (UID: \"7264c907-f536-4719-905a-2bbda9e3c336\") " pod="openshift-ovn-kubernetes/ovnkube-node-js8cc"
May 26 11:24:39.756855 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.756653    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/977e7725-0a14-4446-a572-75ba05e7b12e-host-var-lib-cni-bin\") pod \"multus-fq6hb\" (UID: \"977e7725-0a14-4446-a572-75ba05e7b12e\") " pod="openshift-multus/multus-fq6hb"
May 26 11:24:39.756855 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.756678    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7264c907-f536-4719-905a-2bbda9e3c336-var-lib-openvswitch\") pod \"ovnkube-node-js8cc\" (UID: \"7264c907-f536-4719-905a-2bbda9e3c336\") " pod="openshift-ovn-kubernetes/ovnkube-node-js8cc"
May 26 11:24:39.756855 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.756698    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7264c907-f536-4719-905a-2bbda9e3c336-node-log\") pod \"ovnkube-node-js8cc\" (UID: \"7264c907-f536-4719-905a-2bbda9e3c336\") " pod="openshift-ovn-kubernetes/ovnkube-node-js8cc"
May 26 11:24:39.756855 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.756713    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7264c907-f536-4719-905a-2bbda9e3c336-log-socket\") pod \"ovnkube-node-js8cc\" (UID: \"7264c907-f536-4719-905a-2bbda9e3c336\") " pod="openshift-ovn-kubernetes/ovnkube-node-js8cc"
May 26 11:24:39.756855 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.756725    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7264c907-f536-4719-905a-2bbda9e3c336-host-run-ovn-kubernetes\") pod \"ovnkube-node-js8cc\" (UID: \"7264c907-f536-4719-905a-2bbda9e3c336\") " pod="openshift-ovn-kubernetes/ovnkube-node-js8cc"
May 26 11:24:39.756855 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.756751    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9e35a325-e1c3-4e2f-9ae5-b4120b6f1b60-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vjbr5\" (UID: \"9e35a325-e1c3-4e2f-9ae5-b4120b6f1b60\") " pod="openshift-multus/multus-additional-cni-plugins-vjbr5"
May 26 11:24:39.756855 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.756767    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/977e7725-0a14-4446-a572-75ba05e7b12e-cni-binary-copy\") pod \"multus-fq6hb\" (UID: \"977e7725-0a14-4446-a572-75ba05e7b12e\") " pod="openshift-multus/multus-fq6hb"
May 26 11:24:39.756855 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.756810    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/977e7725-0a14-4446-a572-75ba05e7b12e-multus-socket-dir-parent\") pod \"multus-fq6hb\" (UID: \"977e7725-0a14-4446-a572-75ba05e7b12e\") " pod="openshift-multus/multus-fq6hb"
May 26 11:24:39.756855 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.756819    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/977e7725-0a14-4446-a572-75ba05e7b12e-host-var-lib-cni-bin\") pod \"multus-fq6hb\" (UID: \"977e7725-0a14-4446-a572-75ba05e7b12e\") " pod="openshift-multus/multus-fq6hb"
May 26 11:24:39.756855 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.756824    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9e35a325-e1c3-4e2f-9ae5-b4120b6f1b60-system-cni-dir\") pod \"multus-additional-cni-plugins-vjbr5\" (UID: \"9e35a325-e1c3-4e2f-9ae5-b4120b6f1b60\") " pod="openshift-multus/multus-additional-cni-plugins-vjbr5"
May 26 11:24:39.756855 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.756848    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/977e7725-0a14-4446-a572-75ba05e7b12e-etc-kubernetes\") pod \"multus-fq6hb\" (UID: \"977e7725-0a14-4446-a572-75ba05e7b12e\") " pod="openshift-multus/multus-fq6hb"
May 26 11:24:39.757657 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.756857    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7264c907-f536-4719-905a-2bbda9e3c336-node-log\") pod \"ovnkube-node-js8cc\" (UID: \"7264c907-f536-4719-905a-2bbda9e3c336\") " pod="openshift-ovn-kubernetes/ovnkube-node-js8cc"
May 26 11:24:39.757657 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.756868    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/977e7725-0a14-4446-a572-75ba05e7b12e-host-run-multus-certs\") pod \"multus-fq6hb\" (UID: \"977e7725-0a14-4446-a572-75ba05e7b12e\") " pod="openshift-multus/multus-fq6hb"
May 26 11:24:39.757657 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.756879    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/977e7725-0a14-4446-a572-75ba05e7b12e-host-var-lib-kubelet\") pod \"multus-fq6hb\" (UID: \"977e7725-0a14-4446-a572-75ba05e7b12e\") " pod="openshift-multus/multus-fq6hb"
May 26 11:24:39.757657 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.756904    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/16cccd56-9a2f-44f1-9571-664d5cde0cd7-etc-selinux\") pod \"aws-ebs-csi-driver-node-6g66v\" (UID: \"16cccd56-9a2f-44f1-9571-664d5cde0cd7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6g66v"
May 26 11:24:39.757657 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.756905    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/977e7725-0a14-4446-a572-75ba05e7b12e-hostroot\") pod \"multus-fq6hb\" (UID: \"977e7725-0a14-4446-a572-75ba05e7b12e\") " pod="openshift-multus/multus-fq6hb"
May 26 11:24:39.757657 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.756914    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7264c907-f536-4719-905a-2bbda9e3c336-host-run-ovn-kubernetes\") pod \"ovnkube-node-js8cc\" (UID: \"7264c907-f536-4719-905a-2bbda9e3c336\") " pod="openshift-ovn-kubernetes/ovnkube-node-js8cc"
May 26 11:24:39.757657 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.756769    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/9e35a325-e1c3-4e2f-9ae5-b4120b6f1b60-os-release\") pod \"multus-additional-cni-plugins-vjbr5\" (UID: \"9e35a325-e1c3-4e2f-9ae5-b4120b6f1b60\") " pod="openshift-multus/multus-additional-cni-plugins-vjbr5"
May 26 11:24:39.757657 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.756944    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7264c907-f536-4719-905a-2bbda9e3c336-run-systemd\") pod \"ovnkube-node-js8cc\" (UID: \"7264c907-f536-4719-905a-2bbda9e3c336\") " pod="openshift-ovn-kubernetes/ovnkube-node-js8cc"
May 26 11:24:39.757657 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.756956    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7264c907-f536-4719-905a-2bbda9e3c336-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-js8cc\" (UID: \"7264c907-f536-4719-905a-2bbda9e3c336\") " pod="openshift-ovn-kubernetes/ovnkube-node-js8cc"
May 26 11:24:39.757657 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.756957    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-69l4f\" (UniqueName: \"kubernetes.io/projected/dede2966-ea61-4174-b667-c6fb46152d91-kube-api-access-69l4f\") pod \"network-metrics-daemon-k7jgt\" (UID: \"dede2966-ea61-4174-b667-c6fb46152d91\") " pod="openshift-multus/network-metrics-daemon-k7jgt"
May 26 11:24:39.757657 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.756999    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-7gbx6\" (UniqueName: \"kubernetes.io/projected/e87b53a4-1dca-46b1-bbd4-e484919093c4-kube-api-access-7gbx6\") pod \"network-check-target-cc5xb\" (UID: \"e87b53a4-1dca-46b1-bbd4-e484919093c4\") " pod="openshift-network-diagnostics/network-check-target-cc5xb"
May 26 11:24:39.757657 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.757026    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/fa96a484-9200-4281-a4e2-30c04f5f8b04-konnectivity-ca\") pod \"konnectivity-agent-n4h7j\" (UID: \"fa96a484-9200-4281-a4e2-30c04f5f8b04\") " pod="kube-system/konnectivity-agent-n4h7j"
May 26 11:24:39.757657 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.757054    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/977e7725-0a14-4446-a572-75ba05e7b12e-host-run-netns\") pod \"multus-fq6hb\" (UID: \"977e7725-0a14-4446-a572-75ba05e7b12e\") " pod="openshift-multus/multus-fq6hb"
May 26 11:24:39.757657 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.757079    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/977e7725-0a14-4446-a572-75ba05e7b12e-multus-daemon-config\") pod \"multus-fq6hb\" (UID: \"977e7725-0a14-4446-a572-75ba05e7b12e\") " pod="openshift-multus/multus-fq6hb"
May 26 11:24:39.757657 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.757103    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7264c907-f536-4719-905a-2bbda9e3c336-ovnkube-config\") pod \"ovnkube-node-js8cc\" (UID: \"7264c907-f536-4719-905a-2bbda9e3c336\") " pod="openshift-ovn-kubernetes/ovnkube-node-js8cc"
May 26 11:24:39.757657 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.757128    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-lkxzs\" (UniqueName: \"kubernetes.io/projected/7264c907-f536-4719-905a-2bbda9e3c336-kube-api-access-lkxzs\") pod \"ovnkube-node-js8cc\" (UID: \"7264c907-f536-4719-905a-2bbda9e3c336\") " pod="openshift-ovn-kubernetes/ovnkube-node-js8cc"
May 26 11:24:39.757657 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.757155    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/977e7725-0a14-4446-a572-75ba05e7b12e-system-cni-dir\") pod \"multus-fq6hb\" (UID: \"977e7725-0a14-4446-a572-75ba05e7b12e\") " pod="openshift-multus/multus-fq6hb"
May 26 11:24:39.758309 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.757180    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7264c907-f536-4719-905a-2bbda9e3c336-run-openvswitch\") pod \"ovnkube-node-js8cc\" (UID: \"7264c907-f536-4719-905a-2bbda9e3c336\") " pod="openshift-ovn-kubernetes/ovnkube-node-js8cc"
May 26 11:24:39.758309 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.757220    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9e35a325-e1c3-4e2f-9ae5-b4120b6f1b60-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vjbr5\" (UID: \"9e35a325-e1c3-4e2f-9ae5-b4120b6f1b60\") " pod="openshift-multus/multus-additional-cni-plugins-vjbr5"
May 26 11:24:39.758309 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.757223    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9e35a325-e1c3-4e2f-9ae5-b4120b6f1b60-cni-binary-copy\") pod \"multus-additional-cni-plugins-vjbr5\" (UID: \"9e35a325-e1c3-4e2f-9ae5-b4120b6f1b60\") " pod="openshift-multus/multus-additional-cni-plugins-vjbr5"
May 26 11:24:39.758309 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.757268    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/977e7725-0a14-4446-a572-75ba05e7b12e-host-run-netns\") pod \"multus-fq6hb\" (UID: \"977e7725-0a14-4446-a572-75ba05e7b12e\") " pod="openshift-multus/multus-fq6hb"
May 26 11:24:39.758309 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.757273    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-6kgdk\" (UniqueName: \"kubernetes.io/projected/9e35a325-e1c3-4e2f-9ae5-b4120b6f1b60-kube-api-access-6kgdk\") pod \"multus-additional-cni-plugins-vjbr5\" (UID: \"9e35a325-e1c3-4e2f-9ae5-b4120b6f1b60\") " pod="openshift-multus/multus-additional-cni-plugins-vjbr5"
May 26 11:24:39.758309 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.757308    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/977e7725-0a14-4446-a572-75ba05e7b12e-cnibin\") pod \"multus-fq6hb\" (UID: \"977e7725-0a14-4446-a572-75ba05e7b12e\") " pod="openshift-multus/multus-fq6hb"
May 26 11:24:39.758309 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.757332    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/977e7725-0a14-4446-a572-75ba05e7b12e-multus-conf-dir\") pod \"multus-fq6hb\" (UID: \"977e7725-0a14-4446-a572-75ba05e7b12e\") " pod="openshift-multus/multus-fq6hb"
May 26 11:24:39.758309 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.757356    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-gsjth\" (UniqueName: \"kubernetes.io/projected/977e7725-0a14-4446-a572-75ba05e7b12e-kube-api-access-gsjth\") pod \"multus-fq6hb\" (UID: \"977e7725-0a14-4446-a572-75ba05e7b12e\") " pod="openshift-multus/multus-fq6hb"
May 26 11:24:39.758309 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.757384    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7264c907-f536-4719-905a-2bbda9e3c336-systemd-units\") pod \"ovnkube-node-js8cc\" (UID: \"7264c907-f536-4719-905a-2bbda9e3c336\") " pod="openshift-ovn-kubernetes/ovnkube-node-js8cc"
May 26 11:24:39.758309 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.757431    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/16cccd56-9a2f-44f1-9571-664d5cde0cd7-registration-dir\") pod \"aws-ebs-csi-driver-node-6g66v\" (UID: \"16cccd56-9a2f-44f1-9571-664d5cde0cd7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6g66v"
May 26 11:24:39.758309 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.757455    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7264c907-f536-4719-905a-2bbda9e3c336-host-kubelet\") pod \"ovnkube-node-js8cc\" (UID: \"7264c907-f536-4719-905a-2bbda9e3c336\") " pod="openshift-ovn-kubernetes/ovnkube-node-js8cc"
May 26 11:24:39.758309 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.757480    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7264c907-f536-4719-905a-2bbda9e3c336-etc-openvswitch\") pod \"ovnkube-node-js8cc\" (UID: \"7264c907-f536-4719-905a-2bbda9e3c336\") " pod="openshift-ovn-kubernetes/ovnkube-node-js8cc"
May 26 11:24:39.758309 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.757507    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/16cccd56-9a2f-44f1-9571-664d5cde0cd7-device-dir\") pod \"aws-ebs-csi-driver-node-6g66v\" (UID: \"16cccd56-9a2f-44f1-9571-664d5cde0cd7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6g66v"
May 26 11:24:39.758309 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.757519    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/fa96a484-9200-4281-a4e2-30c04f5f8b04-konnectivity-ca\") pod \"konnectivity-agent-n4h7j\" (UID: \"fa96a484-9200-4281-a4e2-30c04f5f8b04\") " pod="kube-system/konnectivity-agent-n4h7j"
May 26 11:24:39.758309 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.757531    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/16cccd56-9a2f-44f1-9571-664d5cde0cd7-sys-fs\") pod \"aws-ebs-csi-driver-node-6g66v\" (UID: \"16cccd56-9a2f-44f1-9571-664d5cde0cd7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6g66v"
May 26 11:24:39.758309 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.757582    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/977e7725-0a14-4446-a572-75ba05e7b12e-cnibin\") pod \"multus-fq6hb\" (UID: \"977e7725-0a14-4446-a572-75ba05e7b12e\") " pod="openshift-multus/multus-fq6hb"
May 26 11:24:39.758309 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.757634    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7264c907-f536-4719-905a-2bbda9e3c336-host-kubelet\") pod \"ovnkube-node-js8cc\" (UID: \"7264c907-f536-4719-905a-2bbda9e3c336\") " pod="openshift-ovn-kubernetes/ovnkube-node-js8cc"
May 26 11:24:39.759116 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.757657    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/977e7725-0a14-4446-a572-75ba05e7b12e-multus-conf-dir\") pod \"multus-fq6hb\" (UID: \"977e7725-0a14-4446-a572-75ba05e7b12e\") " pod="openshift-multus/multus-fq6hb"
May 26 11:24:39.759116 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.757673    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7264c907-f536-4719-905a-2bbda9e3c336-etc-openvswitch\") pod \"ovnkube-node-js8cc\" (UID: \"7264c907-f536-4719-905a-2bbda9e3c336\") " pod="openshift-ovn-kubernetes/ovnkube-node-js8cc"
May 26 11:24:39.759116 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.757715    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/16cccd56-9a2f-44f1-9571-664d5cde0cd7-registration-dir\") pod \"aws-ebs-csi-driver-node-6g66v\" (UID: \"16cccd56-9a2f-44f1-9571-664d5cde0cd7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6g66v"
May 26 11:24:39.759116 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.757735    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/16cccd56-9a2f-44f1-9571-664d5cde0cd7-device-dir\") pod \"aws-ebs-csi-driver-node-6g66v\" (UID: \"16cccd56-9a2f-44f1-9571-664d5cde0cd7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6g66v"
May 26 11:24:39.759116 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.757779    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7264c907-f536-4719-905a-2bbda9e3c336-run-openvswitch\") pod \"ovnkube-node-js8cc\" (UID: \"7264c907-f536-4719-905a-2bbda9e3c336\") " pod="openshift-ovn-kubernetes/ovnkube-node-js8cc"
May 26 11:24:39.759116 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.757780    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/977e7725-0a14-4446-a572-75ba05e7b12e-system-cni-dir\") pod \"multus-fq6hb\" (UID: \"977e7725-0a14-4446-a572-75ba05e7b12e\") " pod="openshift-multus/multus-fq6hb"
May 26 11:24:39.759116 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.757839    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/16cccd56-9a2f-44f1-9571-664d5cde0cd7-sys-fs\") pod \"aws-ebs-csi-driver-node-6g66v\" (UID: \"16cccd56-9a2f-44f1-9571-664d5cde0cd7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6g66v"
May 26 11:24:39.759116 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.757876    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7264c907-f536-4719-905a-2bbda9e3c336-systemd-units\") pod \"ovnkube-node-js8cc\" (UID: \"7264c907-f536-4719-905a-2bbda9e3c336\") " pod="openshift-ovn-kubernetes/ovnkube-node-js8cc"
May 26 11:24:39.759116 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.758046    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/977e7725-0a14-4446-a572-75ba05e7b12e-multus-daemon-config\") pod \"multus-fq6hb\" (UID: \"977e7725-0a14-4446-a572-75ba05e7b12e\") " pod="openshift-multus/multus-fq6hb"
May 26 11:24:39.759116 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.758180    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7264c907-f536-4719-905a-2bbda9e3c336-ovnkube-config\") pod \"ovnkube-node-js8cc\" (UID: \"7264c907-f536-4719-905a-2bbda9e3c336\") " pod="openshift-ovn-kubernetes/ovnkube-node-js8cc"
May 26 11:24:39.759116 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.758669    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9e35a325-e1c3-4e2f-9ae5-b4120b6f1b60-cni-binary-copy\") pod \"multus-additional-cni-plugins-vjbr5\" (UID: \"9e35a325-e1c3-4e2f-9ae5-b4120b6f1b60\") " pod="openshift-multus/multus-additional-cni-plugins-vjbr5"
May 26 11:24:39.759116 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.758956    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/fa96a484-9200-4281-a4e2-30c04f5f8b04-agent-certs\") pod \"konnectivity-agent-n4h7j\" (UID: \"fa96a484-9200-4281-a4e2-30c04f5f8b04\") " pod="kube-system/konnectivity-agent-n4h7j"
May 26 11:24:39.759805 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.759481    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7264c907-f536-4719-905a-2bbda9e3c336-ovn-node-metrics-cert\") pod \"ovnkube-node-js8cc\" (UID: \"7264c907-f536-4719-905a-2bbda9e3c336\") " pod="openshift-ovn-kubernetes/ovnkube-node-js8cc"
May 26 11:24:39.775258 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.775239    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjc5p\" (UniqueName: \"kubernetes.io/projected/16cccd56-9a2f-44f1-9571-664d5cde0cd7-kube-api-access-jjc5p\") pod \"aws-ebs-csi-driver-node-6g66v\" (UID: \"16cccd56-9a2f-44f1-9571-664d5cde0cd7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6g66v"
May 26 11:24:39.777600 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.777577    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-69l4f\" (UniqueName: \"kubernetes.io/projected/dede2966-ea61-4174-b667-c6fb46152d91-kube-api-access-69l4f\") pod \"network-metrics-daemon-k7jgt\" (UID: \"dede2966-ea61-4174-b667-c6fb46152d91\") " pod="openshift-multus/network-metrics-daemon-k7jgt"
May 26 11:24:39.780808 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.780789    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkxzs\" (UniqueName: \"kubernetes.io/projected/7264c907-f536-4719-905a-2bbda9e3c336-kube-api-access-lkxzs\") pod \"ovnkube-node-js8cc\" (UID: \"7264c907-f536-4719-905a-2bbda9e3c336\") " pod="openshift-ovn-kubernetes/ovnkube-node-js8cc"
May 26 11:24:39.787694 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:24:39.787671    2572 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 26 11:24:39.787694 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:24:39.787691    2572 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 26 11:24:39.787858 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:24:39.787705    2572 projected.go:194] Error preparing data for projected volume kube-api-access-7gbx6 for pod openshift-network-diagnostics/network-check-target-cc5xb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 26 11:24:39.787858 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:24:39.787778    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e87b53a4-1dca-46b1-bbd4-e484919093c4-kube-api-access-7gbx6 podName:e87b53a4-1dca-46b1-bbd4-e484919093c4 nodeName:}" failed. No retries permitted until 2026-05-26 11:24:40.287760339 +0000 UTC m=+3.189970648 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-7gbx6" (UniqueName: "kubernetes.io/projected/e87b53a4-1dca-46b1-bbd4-e484919093c4-kube-api-access-7gbx6") pod "network-check-target-cc5xb" (UID: "e87b53a4-1dca-46b1-bbd4-e484919093c4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 26 11:24:39.791057 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.791038    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kgdk\" (UniqueName: \"kubernetes.io/projected/9e35a325-e1c3-4e2f-9ae5-b4120b6f1b60-kube-api-access-6kgdk\") pod \"multus-additional-cni-plugins-vjbr5\" (UID: \"9e35a325-e1c3-4e2f-9ae5-b4120b6f1b60\") " pod="openshift-multus/multus-additional-cni-plugins-vjbr5"
May 26 11:24:39.792469 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.792449    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gsjth\" (UniqueName: \"kubernetes.io/projected/977e7725-0a14-4446-a572-75ba05e7b12e-kube-api-access-gsjth\") pod \"multus-fq6hb\" (UID: \"977e7725-0a14-4446-a572-75ba05e7b12e\") " pod="openshift-multus/multus-fq6hb"
May 26 11:24:39.845438 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.845335    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-4clzr"
May 26 11:24:39.853735 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.853716    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-s9s2b"
May 26 11:24:39.862431 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.862411    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-gzwdn"
May 26 11:24:39.867869 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.867850    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-qvkjq"
May 26 11:24:39.873024 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.873008    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-vjbr5"
May 26 11:24:39.879627 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.879610    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-n4h7j"
May 26 11:24:39.885425 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.885411    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6g66v"
May 26 11:24:39.891844 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.891828    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-fq6hb"
May 26 11:24:39.897169 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:39.897150    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-js8cc"
May 26 11:24:40.161834 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:40.161793    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7264c907_f536_4719_905a_2bbda9e3c336.slice/crio-e31da5f83b26823ceef33c7fcbaeff63c4ed87439efb060650bbf9766c5e5195 WatchSource:0}: Error finding container e31da5f83b26823ceef33c7fcbaeff63c4ed87439efb060650bbf9766c5e5195: Status 404 returned error can't find the container with id e31da5f83b26823ceef33c7fcbaeff63c4ed87439efb060650bbf9766c5e5195
May 26 11:24:40.163328 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:40.163265    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3603f4d6_abda_490f_aa33_4c3f5eb284f2.slice/crio-8312e40d03e18a37c0edf32648bcdeb346371403cedefb91e89137acdb46b953 WatchSource:0}: Error finding container 8312e40d03e18a37c0edf32648bcdeb346371403cedefb91e89137acdb46b953: Status 404 returned error can't find the container with id 8312e40d03e18a37c0edf32648bcdeb346371403cedefb91e89137acdb46b953
May 26 11:24:40.164235 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:40.164105    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e35a325_e1c3_4e2f_9ae5_b4120b6f1b60.slice/crio-e41be6fe357a9219c753c2d1cf1ce4f5c4f39d9e370c4dc595d13b41a6dd8d70 WatchSource:0}: Error finding container e41be6fe357a9219c753c2d1cf1ce4f5c4f39d9e370c4dc595d13b41a6dd8d70: Status 404 returned error can't find the container with id e41be6fe357a9219c753c2d1cf1ce4f5c4f39d9e370c4dc595d13b41a6dd8d70
May 26 11:24:40.189609 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:40.189570    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16cccd56_9a2f_44f1_9571_664d5cde0cd7.slice/crio-4622bbeabac31f2690b9d0eb5119bc4a6ad87d8843fc0f429c465fe13a8fdbe7 WatchSource:0}: Error finding container 4622bbeabac31f2690b9d0eb5119bc4a6ad87d8843fc0f429c465fe13a8fdbe7: Status 404 returned error can't find the container with id 4622bbeabac31f2690b9d0eb5119bc4a6ad87d8843fc0f429c465fe13a8fdbe7
May 26 11:24:40.190492 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:40.190451    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod977e7725_0a14_4446_a572_75ba05e7b12e.slice/crio-bd2cf9543ae6a6f5b5a136806ec7320c0b16c3e44af13dd4678a30ae7ab98a3b WatchSource:0}: Error finding container bd2cf9543ae6a6f5b5a136806ec7320c0b16c3e44af13dd4678a30ae7ab98a3b: Status 404 returned error can't find the container with id bd2cf9543ae6a6f5b5a136806ec7320c0b16c3e44af13dd4678a30ae7ab98a3b
May 26 11:24:40.191648 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:40.191624    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91fba38c_504f_41ac_8b14_f8e6f9085f0b.slice/crio-9da2268770568c9c0091fe368007d2e24d55c918c643b791e1e73e918b49d565 WatchSource:0}: Error finding container 9da2268770568c9c0091fe368007d2e24d55c918c643b791e1e73e918b49d565: Status 404 returned error can't find the container with id 9da2268770568c9c0091fe368007d2e24d55c918c643b791e1e73e918b49d565
May 26 11:24:40.193473 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:40.193073    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa96a484_9200_4281_a4e2_30c04f5f8b04.slice/crio-a9e2254cd6b0ee800bde021fa722dd5351edbc6f28487a557bddbaf5d0dae4b8 WatchSource:0}: Error finding container a9e2254cd6b0ee800bde021fa722dd5351edbc6f28487a557bddbaf5d0dae4b8: Status 404 returned error can't find the container with id a9e2254cd6b0ee800bde021fa722dd5351edbc6f28487a557bddbaf5d0dae4b8
May 26 11:24:40.193473 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:24:40.193443    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod240a9693_e4b9_485c_a138_6c3eb7f6734a.slice/crio-56b15ff93179c060ff6e9ce555167e60cdf4e3b34e3c3685324150f7739babe7 WatchSource:0}: Error finding container 56b15ff93179c060ff6e9ce555167e60cdf4e3b34e3c3685324150f7739babe7: Status 404 returned error can't find the container with id 56b15ff93179c060ff6e9ce555167e60cdf4e3b34e3c3685324150f7739babe7
May 26 11:24:40.261312 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:40.261286    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dede2966-ea61-4174-b667-c6fb46152d91-metrics-certs\") pod \"network-metrics-daemon-k7jgt\" (UID: \"dede2966-ea61-4174-b667-c6fb46152d91\") " pod="openshift-multus/network-metrics-daemon-k7jgt"
May 26 11:24:40.261418 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:24:40.261406    2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 26 11:24:40.261484 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:24:40.261473    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dede2966-ea61-4174-b667-c6fb46152d91-metrics-certs podName:dede2966-ea61-4174-b667-c6fb46152d91 nodeName:}" failed. No retries permitted until 2026-05-26 11:24:41.261453102 +0000 UTC m=+4.163663410 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dede2966-ea61-4174-b667-c6fb46152d91-metrics-certs") pod "network-metrics-daemon-k7jgt" (UID: "dede2966-ea61-4174-b667-c6fb46152d91") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 26 11:24:40.362537 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:40.362505    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-7gbx6\" (UniqueName: \"kubernetes.io/projected/e87b53a4-1dca-46b1-bbd4-e484919093c4-kube-api-access-7gbx6\") pod \"network-check-target-cc5xb\" (UID: \"e87b53a4-1dca-46b1-bbd4-e484919093c4\") " pod="openshift-network-diagnostics/network-check-target-cc5xb"
May 26 11:24:40.362724 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:24:40.362702    2572 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 26 11:24:40.362785 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:24:40.362732    2572 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 26 11:24:40.362785 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:24:40.362747    2572 projected.go:194] Error preparing data for projected volume kube-api-access-7gbx6 for pod openshift-network-diagnostics/network-check-target-cc5xb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 26 11:24:40.362879 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:24:40.362816    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e87b53a4-1dca-46b1-bbd4-e484919093c4-kube-api-access-7gbx6 podName:e87b53a4-1dca-46b1-bbd4-e484919093c4 nodeName:}" failed. No retries permitted until 2026-05-26 11:24:41.36279662 +0000 UTC m=+4.265006928 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-7gbx6" (UniqueName: "kubernetes.io/projected/e87b53a4-1dca-46b1-bbd4-e484919093c4-kube-api-access-7gbx6") pod "network-check-target-cc5xb" (UID: "e87b53a4-1dca-46b1-bbd4-e484919093c4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 26 11:24:40.590426 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:40.590354    2572 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-05-25 11:19:38 +0000 UTC" deadline="2027-12-07 15:12:41.870458554 +0000 UTC"
May 26 11:24:40.590426 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:40.590392    2572 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13443h48m1.280070423s"
May 26 11:24:40.672156 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:40.671568    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k7jgt"
May 26 11:24:40.672156 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:24:40.671700    2572 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k7jgt" podUID="dede2966-ea61-4174-b667-c6fb46152d91"
May 26 11:24:40.702945 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:40.702845    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-qvkjq" event={"ID":"91fba38c-504f-41ac-8b14-f8e6f9085f0b","Type":"ContainerStarted","Data":"9da2268770568c9c0091fe368007d2e24d55c918c643b791e1e73e918b49d565"}
May 26 11:24:40.714171 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:40.714103    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-fq6hb" event={"ID":"977e7725-0a14-4446-a572-75ba05e7b12e","Type":"ContainerStarted","Data":"bd2cf9543ae6a6f5b5a136806ec7320c0b16c3e44af13dd4678a30ae7ab98a3b"}
May 26 11:24:40.716885 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:40.716838    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-4clzr" event={"ID":"024ce274-4936-49fa-b084-ec86c46d34ed","Type":"ContainerStarted","Data":"901b50f33d45ad7ae877314f6e5f36b08cc3dc1c9181efbfa5c6248fdd755358"}
May 26 11:24:40.727865 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:40.727835    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-js8cc" event={"ID":"7264c907-f536-4719-905a-2bbda9e3c336","Type":"ContainerStarted","Data":"e31da5f83b26823ceef33c7fcbaeff63c4ed87439efb060650bbf9766c5e5195"}
May 26 11:24:40.734233 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:40.733489    2572 generic.go:358] "Generic (PLEG): container finished" podID="4d5d5c010d8065f0e4964c93d9cb64bf" containerID="ef05e605263cb1312351eef8c39e303ea03e2cae50e4c038fbf0ba85ccfa6025" exitCode=0
May 26 11:24:40.734233 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:40.734003    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-194.ec2.internal" event={"ID":"4d5d5c010d8065f0e4964c93d9cb64bf","Type":"ContainerDied","Data":"ef05e605263cb1312351eef8c39e303ea03e2cae50e4c038fbf0ba85ccfa6025"}
May 26 11:24:40.739890 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:40.739861    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6g66v" event={"ID":"16cccd56-9a2f-44f1-9571-664d5cde0cd7","Type":"ContainerStarted","Data":"4622bbeabac31f2690b9d0eb5119bc4a6ad87d8843fc0f429c465fe13a8fdbe7"}
May 26 11:24:40.760882 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:40.760815    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-s9s2b" event={"ID":"240a9693-e4b9-485c-a138-6c3eb7f6734a","Type":"ContainerStarted","Data":"56b15ff93179c060ff6e9ce555167e60cdf4e3b34e3c3685324150f7739babe7"}
May 26 11:24:40.773068 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:40.773033    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-n4h7j" event={"ID":"fa96a484-9200-4281-a4e2-30c04f5f8b04","Type":"ContainerStarted","Data":"a9e2254cd6b0ee800bde021fa722dd5351edbc6f28487a557bddbaf5d0dae4b8"}
May 26 11:24:40.788072 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:40.788041    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vjbr5" event={"ID":"9e35a325-e1c3-4e2f-9ae5-b4120b6f1b60","Type":"ContainerStarted","Data":"e41be6fe357a9219c753c2d1cf1ce4f5c4f39d9e370c4dc595d13b41a6dd8d70"}
May 26 11:24:40.796165 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:40.796107    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-gzwdn" event={"ID":"3603f4d6-abda-490f-aa33-4c3f5eb284f2","Type":"ContainerStarted","Data":"8312e40d03e18a37c0edf32648bcdeb346371403cedefb91e89137acdb46b953"}
May 26 11:24:40.810671 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:40.810596    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-194.ec2.internal" event={"ID":"4e9c0f443794a935b3209de8b5efc30d","Type":"ContainerStarted","Data":"e3926b23bc7d25c05a31cea8bedcf3c236cd9c2bb03d44cff07b633e276e1c61"}
May 26 11:24:41.269988 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:41.269343    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dede2966-ea61-4174-b667-c6fb46152d91-metrics-certs\") pod \"network-metrics-daemon-k7jgt\" (UID: \"dede2966-ea61-4174-b667-c6fb46152d91\") " pod="openshift-multus/network-metrics-daemon-k7jgt"
May 26 11:24:41.269988 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:24:41.269498    2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 26 11:24:41.269988 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:24:41.269563    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dede2966-ea61-4174-b667-c6fb46152d91-metrics-certs podName:dede2966-ea61-4174-b667-c6fb46152d91 nodeName:}" failed. No retries permitted until 2026-05-26 11:24:43.26954453 +0000 UTC m=+6.171754861 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dede2966-ea61-4174-b667-c6fb46152d91-metrics-certs") pod "network-metrics-daemon-k7jgt" (UID: "dede2966-ea61-4174-b667-c6fb46152d91") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 26 11:24:41.370076 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:41.370005    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-7gbx6\" (UniqueName: \"kubernetes.io/projected/e87b53a4-1dca-46b1-bbd4-e484919093c4-kube-api-access-7gbx6\") pod \"network-check-target-cc5xb\" (UID: \"e87b53a4-1dca-46b1-bbd4-e484919093c4\") " pod="openshift-network-diagnostics/network-check-target-cc5xb"
May 26 11:24:41.370319 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:24:41.370203    2572 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 26 11:24:41.370319 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:24:41.370223    2572 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 26 11:24:41.370319 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:24:41.370235    2572 projected.go:194] Error preparing data for projected volume kube-api-access-7gbx6 for pod openshift-network-diagnostics/network-check-target-cc5xb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 26 11:24:41.370319 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:24:41.370298    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e87b53a4-1dca-46b1-bbd4-e484919093c4-kube-api-access-7gbx6 podName:e87b53a4-1dca-46b1-bbd4-e484919093c4 nodeName:}" failed. No retries permitted until 2026-05-26 11:24:43.370280254 +0000 UTC m=+6.272490577 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-7gbx6" (UniqueName: "kubernetes.io/projected/e87b53a4-1dca-46b1-bbd4-e484919093c4-kube-api-access-7gbx6") pod "network-check-target-cc5xb" (UID: "e87b53a4-1dca-46b1-bbd4-e484919093c4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 26 11:24:41.673718 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:41.673641    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cc5xb"
May 26 11:24:41.674145 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:24:41.673785    2572 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cc5xb" podUID="e87b53a4-1dca-46b1-bbd4-e484919093c4"
May 26 11:24:41.842569 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:41.842109    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-194.ec2.internal" event={"ID":"4d5d5c010d8065f0e4964c93d9cb64bf","Type":"ContainerStarted","Data":"714b6d6133f53335af39f9354b6c00e78023d8d2f902fef3eceaf008c3782250"}
May 26 11:24:41.857236 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:41.855331    2572 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-194.ec2.internal" podStartSLOduration=3.855313407 podStartE2EDuration="3.855313407s" podCreationTimestamp="2026-05-26 11:24:38 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 11:24:40.826498788 +0000 UTC m=+3.728709119" watchObservedRunningTime="2026-05-26 11:24:41.855313407 +0000 UTC m=+4.757523735"
May 26 11:24:42.671991 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:42.671477    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k7jgt"
May 26 11:24:42.671991 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:24:42.671632    2572 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k7jgt" podUID="dede2966-ea61-4174-b667-c6fb46152d91"
May 26 11:24:43.287839 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:43.287709    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dede2966-ea61-4174-b667-c6fb46152d91-metrics-certs\") pod \"network-metrics-daemon-k7jgt\" (UID: \"dede2966-ea61-4174-b667-c6fb46152d91\") " pod="openshift-multus/network-metrics-daemon-k7jgt"
May 26 11:24:43.288329 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:24:43.287887    2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 26 11:24:43.288329 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:24:43.287964    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dede2966-ea61-4174-b667-c6fb46152d91-metrics-certs podName:dede2966-ea61-4174-b667-c6fb46152d91 nodeName:}" failed. No retries permitted until 2026-05-26 11:24:47.287943128 +0000 UTC m=+10.190153461 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dede2966-ea61-4174-b667-c6fb46152d91-metrics-certs") pod "network-metrics-daemon-k7jgt" (UID: "dede2966-ea61-4174-b667-c6fb46152d91") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 26 11:24:43.389065 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:43.389035    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-7gbx6\" (UniqueName: \"kubernetes.io/projected/e87b53a4-1dca-46b1-bbd4-e484919093c4-kube-api-access-7gbx6\") pod \"network-check-target-cc5xb\" (UID: \"e87b53a4-1dca-46b1-bbd4-e484919093c4\") " pod="openshift-network-diagnostics/network-check-target-cc5xb"
May 26 11:24:43.389229 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:24:43.389216    2572 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 26 11:24:43.389307 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:24:43.389239    2572 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 26 11:24:43.389307 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:24:43.389252    2572 projected.go:194] Error preparing data for projected volume kube-api-access-7gbx6 for pod openshift-network-diagnostics/network-check-target-cc5xb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 26 11:24:43.389409 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:24:43.389312    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e87b53a4-1dca-46b1-bbd4-e484919093c4-kube-api-access-7gbx6 podName:e87b53a4-1dca-46b1-bbd4-e484919093c4 nodeName:}" failed. No retries permitted until 2026-05-26 11:24:47.389293584 +0000 UTC m=+10.291503907 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-7gbx6" (UniqueName: "kubernetes.io/projected/e87b53a4-1dca-46b1-bbd4-e484919093c4-kube-api-access-7gbx6") pod "network-check-target-cc5xb" (UID: "e87b53a4-1dca-46b1-bbd4-e484919093c4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 26 11:24:43.671128 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:43.671052    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cc5xb"
May 26 11:24:43.671299 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:24:43.671175    2572 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cc5xb" podUID="e87b53a4-1dca-46b1-bbd4-e484919093c4"
May 26 11:24:44.671497 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:44.671461    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k7jgt"
May 26 11:24:44.671954 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:24:44.671620    2572 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k7jgt" podUID="dede2966-ea61-4174-b667-c6fb46152d91"
May 26 11:24:45.671335 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:45.670801    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cc5xb"
May 26 11:24:45.671335 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:24:45.670934    2572 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cc5xb" podUID="e87b53a4-1dca-46b1-bbd4-e484919093c4"
May 26 11:24:46.237959 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:46.237896    2572 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-194.ec2.internal" podStartSLOduration=8.237877613 podStartE2EDuration="8.237877613s" podCreationTimestamp="2026-05-26 11:24:38 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 11:24:41.855469468 +0000 UTC m=+4.757679789" watchObservedRunningTime="2026-05-26 11:24:46.237877613 +0000 UTC m=+9.140087942"
May 26 11:24:46.238427 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:46.238367    2572 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-5jpd7"]
May 26 11:24:46.241388 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:46.241353    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5jpd7"
May 26 11:24:46.241527 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:24:46.241432    2572 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5jpd7" podUID="2ce3eec7-5b62-4e4f-ad7f-ffc1eda0c2d1"
May 26 11:24:46.317713 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:46.317590    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2ce3eec7-5b62-4e4f-ad7f-ffc1eda0c2d1-kubelet-config\") pod \"global-pull-secret-syncer-5jpd7\" (UID: \"2ce3eec7-5b62-4e4f-ad7f-ffc1eda0c2d1\") " pod="kube-system/global-pull-secret-syncer-5jpd7"
May 26 11:24:46.317713 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:46.317639    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2ce3eec7-5b62-4e4f-ad7f-ffc1eda0c2d1-original-pull-secret\") pod \"global-pull-secret-syncer-5jpd7\" (UID: \"2ce3eec7-5b62-4e4f-ad7f-ffc1eda0c2d1\") " pod="kube-system/global-pull-secret-syncer-5jpd7"
May 26 11:24:46.317713 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:46.317673    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2ce3eec7-5b62-4e4f-ad7f-ffc1eda0c2d1-dbus\") pod \"global-pull-secret-syncer-5jpd7\" (UID: \"2ce3eec7-5b62-4e4f-ad7f-ffc1eda0c2d1\") " pod="kube-system/global-pull-secret-syncer-5jpd7"
May 26 11:24:46.419157 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:46.419116    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2ce3eec7-5b62-4e4f-ad7f-ffc1eda0c2d1-kubelet-config\") pod \"global-pull-secret-syncer-5jpd7\" (UID: \"2ce3eec7-5b62-4e4f-ad7f-ffc1eda0c2d1\") " pod="kube-system/global-pull-secret-syncer-5jpd7"
May 26 11:24:46.419336 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:46.419162    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2ce3eec7-5b62-4e4f-ad7f-ffc1eda0c2d1-original-pull-secret\") pod \"global-pull-secret-syncer-5jpd7\" (UID: \"2ce3eec7-5b62-4e4f-ad7f-ffc1eda0c2d1\") " pod="kube-system/global-pull-secret-syncer-5jpd7"
May 26 11:24:46.419336 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:46.419211    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2ce3eec7-5b62-4e4f-ad7f-ffc1eda0c2d1-dbus\") pod \"global-pull-secret-syncer-5jpd7\" (UID: \"2ce3eec7-5b62-4e4f-ad7f-ffc1eda0c2d1\") " pod="kube-system/global-pull-secret-syncer-5jpd7"
May 26 11:24:46.419582 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:46.419463    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2ce3eec7-5b62-4e4f-ad7f-ffc1eda0c2d1-dbus\") pod \"global-pull-secret-syncer-5jpd7\" (UID: \"2ce3eec7-5b62-4e4f-ad7f-ffc1eda0c2d1\") " pod="kube-system/global-pull-secret-syncer-5jpd7"
May 26 11:24:46.419582 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:46.419539    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2ce3eec7-5b62-4e4f-ad7f-ffc1eda0c2d1-kubelet-config\") pod \"global-pull-secret-syncer-5jpd7\" (UID: \"2ce3eec7-5b62-4e4f-ad7f-ffc1eda0c2d1\") " pod="kube-system/global-pull-secret-syncer-5jpd7"
May 26 11:24:46.419712 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:24:46.419580    2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
May 26 11:24:46.419712 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:24:46.419651    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2ce3eec7-5b62-4e4f-ad7f-ffc1eda0c2d1-original-pull-secret podName:2ce3eec7-5b62-4e4f-ad7f-ffc1eda0c2d1 nodeName:}" failed. No retries permitted until 2026-05-26 11:24:46.919631782 +0000 UTC m=+9.821842095 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2ce3eec7-5b62-4e4f-ad7f-ffc1eda0c2d1-original-pull-secret") pod "global-pull-secret-syncer-5jpd7" (UID: "2ce3eec7-5b62-4e4f-ad7f-ffc1eda0c2d1") : object "kube-system"/"original-pull-secret" not registered
May 26 11:24:46.671139 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:46.671010    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k7jgt"
May 26 11:24:46.671323 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:24:46.671158    2572 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k7jgt" podUID="dede2966-ea61-4174-b667-c6fb46152d91"
May 26 11:24:46.924696 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:46.924610    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2ce3eec7-5b62-4e4f-ad7f-ffc1eda0c2d1-original-pull-secret\") pod \"global-pull-secret-syncer-5jpd7\" (UID: \"2ce3eec7-5b62-4e4f-ad7f-ffc1eda0c2d1\") " pod="kube-system/global-pull-secret-syncer-5jpd7"
May 26 11:24:46.924888 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:24:46.924852    2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
May 26 11:24:46.924949 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:24:46.924927    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2ce3eec7-5b62-4e4f-ad7f-ffc1eda0c2d1-original-pull-secret podName:2ce3eec7-5b62-4e4f-ad7f-ffc1eda0c2d1 nodeName:}" failed. No retries permitted until 2026-05-26 11:24:47.924907477 +0000 UTC m=+10.827117800 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2ce3eec7-5b62-4e4f-ad7f-ffc1eda0c2d1-original-pull-secret") pod "global-pull-secret-syncer-5jpd7" (UID: "2ce3eec7-5b62-4e4f-ad7f-ffc1eda0c2d1") : object "kube-system"/"original-pull-secret" not registered
May 26 11:24:47.327531 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:47.327448    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dede2966-ea61-4174-b667-c6fb46152d91-metrics-certs\") pod \"network-metrics-daemon-k7jgt\" (UID: \"dede2966-ea61-4174-b667-c6fb46152d91\") " pod="openshift-multus/network-metrics-daemon-k7jgt"
May 26 11:24:47.327988 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:24:47.327594    2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 26 11:24:47.327988 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:24:47.327655    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dede2966-ea61-4174-b667-c6fb46152d91-metrics-certs podName:dede2966-ea61-4174-b667-c6fb46152d91 nodeName:}" failed. No retries permitted until 2026-05-26 11:24:55.327636458 +0000 UTC m=+18.229846779 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dede2966-ea61-4174-b667-c6fb46152d91-metrics-certs") pod "network-metrics-daemon-k7jgt" (UID: "dede2966-ea61-4174-b667-c6fb46152d91") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 26 11:24:47.428155 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:47.428118    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-7gbx6\" (UniqueName: \"kubernetes.io/projected/e87b53a4-1dca-46b1-bbd4-e484919093c4-kube-api-access-7gbx6\") pod \"network-check-target-cc5xb\" (UID: \"e87b53a4-1dca-46b1-bbd4-e484919093c4\") " pod="openshift-network-diagnostics/network-check-target-cc5xb"
May 26 11:24:47.428334 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:24:47.428299    2572 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 26 11:24:47.428334 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:24:47.428322    2572 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 26 11:24:47.428334 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:24:47.428335    2572 projected.go:194] Error preparing data for projected volume kube-api-access-7gbx6 for pod openshift-network-diagnostics/network-check-target-cc5xb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 26 11:24:47.428504 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:24:47.428400    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e87b53a4-1dca-46b1-bbd4-e484919093c4-kube-api-access-7gbx6 podName:e87b53a4-1dca-46b1-bbd4-e484919093c4 nodeName:}" failed. No retries permitted until 2026-05-26 11:24:55.428382588 +0000 UTC m=+18.330592916 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-7gbx6" (UniqueName: "kubernetes.io/projected/e87b53a4-1dca-46b1-bbd4-e484919093c4-kube-api-access-7gbx6") pod "network-check-target-cc5xb" (UID: "e87b53a4-1dca-46b1-bbd4-e484919093c4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 26 11:24:47.677336 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:47.677260    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cc5xb"
May 26 11:24:47.677887 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:24:47.677860    2572 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cc5xb" podUID="e87b53a4-1dca-46b1-bbd4-e484919093c4"
May 26 11:24:47.678036 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:47.677754    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5jpd7"
May 26 11:24:47.678216 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:24:47.678180    2572 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5jpd7" podUID="2ce3eec7-5b62-4e4f-ad7f-ffc1eda0c2d1"
May 26 11:24:47.933953 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:47.933872    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2ce3eec7-5b62-4e4f-ad7f-ffc1eda0c2d1-original-pull-secret\") pod \"global-pull-secret-syncer-5jpd7\" (UID: \"2ce3eec7-5b62-4e4f-ad7f-ffc1eda0c2d1\") " pod="kube-system/global-pull-secret-syncer-5jpd7"
May 26 11:24:47.934102 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:24:47.934027    2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
May 26 11:24:47.934166 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:24:47.934110    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2ce3eec7-5b62-4e4f-ad7f-ffc1eda0c2d1-original-pull-secret podName:2ce3eec7-5b62-4e4f-ad7f-ffc1eda0c2d1 nodeName:}" failed. No retries permitted until 2026-05-26 11:24:49.934088967 +0000 UTC m=+12.836299278 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2ce3eec7-5b62-4e4f-ad7f-ffc1eda0c2d1-original-pull-secret") pod "global-pull-secret-syncer-5jpd7" (UID: "2ce3eec7-5b62-4e4f-ad7f-ffc1eda0c2d1") : object "kube-system"/"original-pull-secret" not registered
May 26 11:24:48.671178 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:48.671139    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k7jgt"
May 26 11:24:48.671636 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:24:48.671298    2572 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k7jgt" podUID="dede2966-ea61-4174-b667-c6fb46152d91"
May 26 11:24:49.671256 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:49.671222    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5jpd7"
May 26 11:24:49.671675 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:49.671269    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cc5xb"
May 26 11:24:49.671675 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:24:49.671350    2572 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5jpd7" podUID="2ce3eec7-5b62-4e4f-ad7f-ffc1eda0c2d1"
May 26 11:24:49.671675 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:24:49.671469    2572 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cc5xb" podUID="e87b53a4-1dca-46b1-bbd4-e484919093c4"
May 26 11:24:49.949118 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:49.949029    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2ce3eec7-5b62-4e4f-ad7f-ffc1eda0c2d1-original-pull-secret\") pod \"global-pull-secret-syncer-5jpd7\" (UID: \"2ce3eec7-5b62-4e4f-ad7f-ffc1eda0c2d1\") " pod="kube-system/global-pull-secret-syncer-5jpd7"
May 26 11:24:49.949306 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:24:49.949181    2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
May 26 11:24:49.949306 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:24:49.949259    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2ce3eec7-5b62-4e4f-ad7f-ffc1eda0c2d1-original-pull-secret podName:2ce3eec7-5b62-4e4f-ad7f-ffc1eda0c2d1 nodeName:}" failed. No retries permitted until 2026-05-26 11:24:53.949243742 +0000 UTC m=+16.851454056 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2ce3eec7-5b62-4e4f-ad7f-ffc1eda0c2d1-original-pull-secret") pod "global-pull-secret-syncer-5jpd7" (UID: "2ce3eec7-5b62-4e4f-ad7f-ffc1eda0c2d1") : object "kube-system"/"original-pull-secret" not registered
May 26 11:24:50.671362 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:50.671330    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k7jgt"
May 26 11:24:50.671830 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:24:50.671465    2572 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k7jgt" podUID="dede2966-ea61-4174-b667-c6fb46152d91"
May 26 11:24:51.671314 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:51.671277    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cc5xb"
May 26 11:24:51.671502 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:24:51.671413    2572 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cc5xb" podUID="e87b53a4-1dca-46b1-bbd4-e484919093c4"
May 26 11:24:51.671502 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:51.671455    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5jpd7"
May 26 11:24:51.671823 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:24:51.671507    2572 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5jpd7" podUID="2ce3eec7-5b62-4e4f-ad7f-ffc1eda0c2d1"
May 26 11:24:52.670801 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:52.670766    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k7jgt"
May 26 11:24:52.670982 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:24:52.670895    2572 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k7jgt" podUID="dede2966-ea61-4174-b667-c6fb46152d91"
May 26 11:24:53.671365 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:53.671328    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cc5xb"
May 26 11:24:53.671365 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:53.671353    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5jpd7"
May 26 11:24:53.671863 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:24:53.671453    2572 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cc5xb" podUID="e87b53a4-1dca-46b1-bbd4-e484919093c4"
May 26 11:24:53.671863 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:24:53.671587    2572 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5jpd7" podUID="2ce3eec7-5b62-4e4f-ad7f-ffc1eda0c2d1"
May 26 11:24:53.984543 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:53.984424    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2ce3eec7-5b62-4e4f-ad7f-ffc1eda0c2d1-original-pull-secret\") pod \"global-pull-secret-syncer-5jpd7\" (UID: \"2ce3eec7-5b62-4e4f-ad7f-ffc1eda0c2d1\") " pod="kube-system/global-pull-secret-syncer-5jpd7"
May 26 11:24:53.984711 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:24:53.984621    2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
May 26 11:24:53.984711 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:24:53.984693    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2ce3eec7-5b62-4e4f-ad7f-ffc1eda0c2d1-original-pull-secret podName:2ce3eec7-5b62-4e4f-ad7f-ffc1eda0c2d1 nodeName:}" failed. No retries permitted until 2026-05-26 11:25:01.984677969 +0000 UTC m=+24.886888279 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2ce3eec7-5b62-4e4f-ad7f-ffc1eda0c2d1-original-pull-secret") pod "global-pull-secret-syncer-5jpd7" (UID: "2ce3eec7-5b62-4e4f-ad7f-ffc1eda0c2d1") : object "kube-system"/"original-pull-secret" not registered
May 26 11:24:54.670876 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:54.670837    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k7jgt"
May 26 11:24:54.671046 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:24:54.670985    2572 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k7jgt" podUID="dede2966-ea61-4174-b667-c6fb46152d91"
May 26 11:24:55.393449 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:55.393412    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dede2966-ea61-4174-b667-c6fb46152d91-metrics-certs\") pod \"network-metrics-daemon-k7jgt\" (UID: \"dede2966-ea61-4174-b667-c6fb46152d91\") " pod="openshift-multus/network-metrics-daemon-k7jgt"
May 26 11:24:55.393907 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:24:55.393541    2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 26 11:24:55.393907 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:24:55.393600    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dede2966-ea61-4174-b667-c6fb46152d91-metrics-certs podName:dede2966-ea61-4174-b667-c6fb46152d91 nodeName:}" failed. No retries permitted until 2026-05-26 11:25:11.393585374 +0000 UTC m=+34.295795685 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dede2966-ea61-4174-b667-c6fb46152d91-metrics-certs") pod "network-metrics-daemon-k7jgt" (UID: "dede2966-ea61-4174-b667-c6fb46152d91") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 26 11:24:55.493835 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:55.493801    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-7gbx6\" (UniqueName: \"kubernetes.io/projected/e87b53a4-1dca-46b1-bbd4-e484919093c4-kube-api-access-7gbx6\") pod \"network-check-target-cc5xb\" (UID: \"e87b53a4-1dca-46b1-bbd4-e484919093c4\") " pod="openshift-network-diagnostics/network-check-target-cc5xb"
May 26 11:24:55.494004 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:24:55.493984    2572 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 26 11:24:55.494054 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:24:55.494012    2572 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 26 11:24:55.494054 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:24:55.494025    2572 projected.go:194] Error preparing data for projected volume kube-api-access-7gbx6 for pod openshift-network-diagnostics/network-check-target-cc5xb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 26 11:24:55.494162 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:24:55.494076    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e87b53a4-1dca-46b1-bbd4-e484919093c4-kube-api-access-7gbx6 podName:e87b53a4-1dca-46b1-bbd4-e484919093c4 nodeName:}" failed. No retries permitted until 2026-05-26 11:25:11.494062115 +0000 UTC m=+34.396272429 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-7gbx6" (UniqueName: "kubernetes.io/projected/e87b53a4-1dca-46b1-bbd4-e484919093c4-kube-api-access-7gbx6") pod "network-check-target-cc5xb" (UID: "e87b53a4-1dca-46b1-bbd4-e484919093c4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 26 11:24:55.671148 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:55.671043    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5jpd7"
May 26 11:24:55.671314 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:55.671043    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cc5xb"
May 26 11:24:55.671314 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:24:55.671162    2572 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5jpd7" podUID="2ce3eec7-5b62-4e4f-ad7f-ffc1eda0c2d1"
May 26 11:24:55.671314 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:24:55.671274    2572 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cc5xb" podUID="e87b53a4-1dca-46b1-bbd4-e484919093c4"
May 26 11:24:56.671302 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:56.671106    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k7jgt"
May 26 11:24:56.671689 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:24:56.671402    2572 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k7jgt" podUID="dede2966-ea61-4174-b667-c6fb46152d91"
May 26 11:24:57.671419 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:57.671386    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cc5xb"
May 26 11:24:57.671773 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:24:57.671475    2572 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cc5xb" podUID="e87b53a4-1dca-46b1-bbd4-e484919093c4"
May 26 11:24:57.671773 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:57.671553    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5jpd7"
May 26 11:24:57.671773 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:24:57.671638    2572 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5jpd7" podUID="2ce3eec7-5b62-4e4f-ad7f-ffc1eda0c2d1"
May 26 11:24:58.670901 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:58.670745    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k7jgt"
May 26 11:24:58.671009 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:24:58.670990    2572 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k7jgt" podUID="dede2966-ea61-4174-b667-c6fb46152d91"
May 26 11:24:58.877860 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:58.877831    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-qvkjq" event={"ID":"91fba38c-504f-41ac-8b14-f8e6f9085f0b","Type":"ContainerStarted","Data":"c1de4cdd7e71ecf2367aa05b39bca8f371fe4196419b5dd49f7b4cbd95060c16"}
May 26 11:24:58.879115 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:58.879095    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-fq6hb" event={"ID":"977e7725-0a14-4446-a572-75ba05e7b12e","Type":"ContainerStarted","Data":"dd9c9f5d33f3039542c8cc9e65f8266209d268801a8d87fcc1a3ce1030c2b237"}
May 26 11:24:58.880372 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:58.880347    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-4clzr" event={"ID":"024ce274-4936-49fa-b084-ec86c46d34ed","Type":"ContainerStarted","Data":"9226352abea916ea875446989a8ccdbe679cae91e5d5fde8531c6e05a94f3227"}
May 26 11:24:58.882603 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:58.882584    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-js8cc_7264c907-f536-4719-905a-2bbda9e3c336/ovn-acl-logging/0.log"
May 26 11:24:58.882915 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:58.882894    2572 generic.go:358] "Generic (PLEG): container finished" podID="7264c907-f536-4719-905a-2bbda9e3c336" containerID="c2466d41a89b7a36e4e710eb18868a895d6c1fadf830945e10645d9b81510111" exitCode=1
May 26 11:24:58.883006 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:58.882954    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-js8cc" event={"ID":"7264c907-f536-4719-905a-2bbda9e3c336","Type":"ContainerStarted","Data":"2b1113507a4d774bd975ef693691b296ae6eac064c1d43e72c3d68308619e77f"}
May 26 11:24:58.883006 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:58.882993    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-js8cc" event={"ID":"7264c907-f536-4719-905a-2bbda9e3c336","Type":"ContainerStarted","Data":"927dd8f566a532a1a4d6dc47c29336931e4eff7b5069d67743d5fd0b15b451bb"}
May 26 11:24:58.883118 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:58.883009    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-js8cc" event={"ID":"7264c907-f536-4719-905a-2bbda9e3c336","Type":"ContainerStarted","Data":"70fdb1d9f79c48c938759eb68ec486fa6dd48cd9fcb44940f8eef70c3c84070d"}
May 26 11:24:58.883118 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:58.883022    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-js8cc" event={"ID":"7264c907-f536-4719-905a-2bbda9e3c336","Type":"ContainerDied","Data":"c2466d41a89b7a36e4e710eb18868a895d6c1fadf830945e10645d9b81510111"}
May 26 11:24:58.883118 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:58.883037    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-js8cc" event={"ID":"7264c907-f536-4719-905a-2bbda9e3c336","Type":"ContainerStarted","Data":"6d039c41f1313c3bcee9285aa2114d92e939c2b18e6545b43f460728d2cfc414"}
May 26 11:24:58.884242 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:58.884220    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6g66v" event={"ID":"16cccd56-9a2f-44f1-9571-664d5cde0cd7","Type":"ContainerStarted","Data":"e44eac2ff4e1fe0c7fe366baa9df26503ab2e703bb73cb2eeaa27ba8ec8cf0a0"}
May 26 11:24:58.885557 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:58.885535    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-s9s2b" event={"ID":"240a9693-e4b9-485c-a138-6c3eb7f6734a","Type":"ContainerStarted","Data":"347bf00c706ab137382457b0eba937b9cedc2d4f66de3cac5b93db44b08f76fb"}
May 26 11:24:58.886750 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:58.886724    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-n4h7j" event={"ID":"fa96a484-9200-4281-a4e2-30c04f5f8b04","Type":"ContainerStarted","Data":"b07e81be827b0b703d317fdc612b24c21c510a532478ff934e4146cc13f6c25e"}
May 26 11:24:58.888025 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:58.888006    2572 generic.go:358] "Generic (PLEG): container finished" podID="9e35a325-e1c3-4e2f-9ae5-b4120b6f1b60" containerID="649507c14b89016e06ec8b4bb65c5c8a8451cf98fc249e0990ccaefa61cb891f" exitCode=0
May 26 11:24:58.888094 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:58.888039    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vjbr5" event={"ID":"9e35a325-e1c3-4e2f-9ae5-b4120b6f1b60","Type":"ContainerDied","Data":"649507c14b89016e06ec8b4bb65c5c8a8451cf98fc249e0990ccaefa61cb891f"}
May 26 11:24:58.896611 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:58.896569    2572 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-qvkjq" podStartSLOduration=4.027452224 podStartE2EDuration="21.896557701s" podCreationTimestamp="2026-05-26 11:24:37 +0000 UTC" firstStartedPulling="2026-05-26 11:24:40.196993122 +0000 UTC m=+3.099203438" lastFinishedPulling="2026-05-26 11:24:58.066098608 +0000 UTC m=+20.968308915" observedRunningTime="2026-05-26 11:24:58.896537344 +0000 UTC m=+21.798747674" watchObservedRunningTime="2026-05-26 11:24:58.896557701 +0000 UTC m=+21.798768030"
May 26 11:24:58.913519 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:58.913484    2572 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/multus-fq6hb" podStartSLOduration=3.005385502 podStartE2EDuration="20.91347432s" podCreationTimestamp="2026-05-26 11:24:38 +0000 UTC" firstStartedPulling="2026-05-26 11:24:40.196781704 +0000 UTC m=+3.098992014" lastFinishedPulling="2026-05-26 11:24:58.104870511 +0000 UTC m=+21.007080832" observedRunningTime="2026-05-26 11:24:58.913349162 +0000 UTC m=+21.815559491" watchObservedRunningTime="2026-05-26 11:24:58.91347432 +0000 UTC m=+21.815684648"
May 26 11:24:58.934173 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:58.934136    2572 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-image-registry/node-ca-s9s2b" podStartSLOduration=7.903033171 podStartE2EDuration="21.934127766s" podCreationTimestamp="2026-05-26 11:24:37 +0000 UTC" firstStartedPulling="2026-05-26 11:24:40.196856674 +0000 UTC m=+3.099066992" lastFinishedPulling="2026-05-26 11:24:54.227951277 +0000 UTC m=+17.130161587" observedRunningTime="2026-05-26 11:24:58.92912905 +0000 UTC m=+21.831339389" watchObservedRunningTime="2026-05-26 11:24:58.934127766 +0000 UTC m=+21.836338094"
May 26 11:24:58.986023 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:58.985964    2572 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/konnectivity-agent-n4h7j" podStartSLOduration=4.122774317 podStartE2EDuration="21.985946049s" podCreationTimestamp="2026-05-26 11:24:37 +0000 UTC" firstStartedPulling="2026-05-26 11:24:40.197164721 +0000 UTC m=+3.099375035" lastFinishedPulling="2026-05-26 11:24:58.060336449 +0000 UTC m=+20.962546767" observedRunningTime="2026-05-26 11:24:58.952436449 +0000 UTC m=+21.854646779" watchObservedRunningTime="2026-05-26 11:24:58.985946049 +0000 UTC m=+21.888156379"
May 26 11:24:58.986452 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:58.986421    2572 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-dns/node-resolver-4clzr" podStartSLOduration=4.114619101 podStartE2EDuration="21.986410014s" podCreationTimestamp="2026-05-26 11:24:37 +0000 UTC" firstStartedPulling="2026-05-26 11:24:40.188618836 +0000 UTC m=+3.090829151" lastFinishedPulling="2026-05-26 11:24:58.06040975 +0000 UTC m=+20.962620064" observedRunningTime="2026-05-26 11:24:58.980914454 +0000 UTC m=+21.883124783" watchObservedRunningTime="2026-05-26 11:24:58.986410014 +0000 UTC m=+21.888620344"
May 26 11:24:59.416493 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:59.416452    2572 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
May 26 11:24:59.610333 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:59.610106    2572 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-05-26T11:24:59.416471779Z","UUID":"5ad01dba-81ae-471f-8e8a-c38b4635562c","Handler":null,"Name":"","Endpoint":""}
May 26 11:24:59.614010 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:59.613990    2572 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
May 26 11:24:59.614010 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:59.614014    2572 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
May 26 11:24:59.670763 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:59.670736    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cc5xb"
May 26 11:24:59.670905 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:24:59.670860    2572 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cc5xb" podUID="e87b53a4-1dca-46b1-bbd4-e484919093c4"
May 26 11:24:59.671274 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:59.670888    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5jpd7"
May 26 11:24:59.671568 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:24:59.671542    2572 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5jpd7" podUID="2ce3eec7-5b62-4e4f-ad7f-ffc1eda0c2d1"
May 26 11:24:59.894182 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:59.894040    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-js8cc_7264c907-f536-4719-905a-2bbda9e3c336/ovn-acl-logging/0.log"
May 26 11:24:59.894698 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:59.894495    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-js8cc" event={"ID":"7264c907-f536-4719-905a-2bbda9e3c336","Type":"ContainerStarted","Data":"b7180bf42fbed8131035b933c592333c287a338ebc94dc1ab21478109f333e18"}
May 26 11:24:59.896349 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:59.896303    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6g66v" event={"ID":"16cccd56-9a2f-44f1-9571-664d5cde0cd7","Type":"ContainerStarted","Data":"317b52994d0ebf5655db801e7b8e5bfe9a311b7b55e66fc7acfb25d002917da0"}
May 26 11:24:59.897957 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:59.897919    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-gzwdn" event={"ID":"3603f4d6-abda-490f-aa33-4c3f5eb284f2","Type":"ContainerStarted","Data":"27a2743aef6726ed62435305e400791e7db122a6ad0125e62f536942e4d26e7a"}
May 26 11:24:59.930696 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:24:59.930645    2572 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-gzwdn" podStartSLOduration=5.036234525 podStartE2EDuration="22.930628964s" podCreationTimestamp="2026-05-26 11:24:37 +0000 UTC" firstStartedPulling="2026-05-26 11:24:40.165968243 +0000 UTC m=+3.068178563" lastFinishedPulling="2026-05-26 11:24:58.060362681 +0000 UTC m=+20.962573002" observedRunningTime="2026-05-26 11:24:59.930207056 +0000 UTC m=+22.832417387" watchObservedRunningTime="2026-05-26 11:24:59.930628964 +0000 UTC m=+22.832839295"
May 26 11:25:00.670879 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:25:00.670848    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k7jgt"
May 26 11:25:00.671048 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:25:00.670959    2572 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k7jgt" podUID="dede2966-ea61-4174-b667-c6fb46152d91"
May 26 11:25:00.901979 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:25:00.901934    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6g66v" event={"ID":"16cccd56-9a2f-44f1-9571-664d5cde0cd7","Type":"ContainerStarted","Data":"2e3753ac95ec05f7f445f33f436c9f4492d22cfd184bea03dc1d8b40ca1f5775"}
May 26 11:25:01.670510 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:25:01.670474    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5jpd7"
May 26 11:25:01.670681 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:25:01.670607    2572 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5jpd7" podUID="2ce3eec7-5b62-4e4f-ad7f-ffc1eda0c2d1"
May 26 11:25:01.670749 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:25:01.670708    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cc5xb"
May 26 11:25:01.670870 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:25:01.670845    2572 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cc5xb" podUID="e87b53a4-1dca-46b1-bbd4-e484919093c4"
May 26 11:25:01.906874 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:25:01.906842    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-js8cc_7264c907-f536-4719-905a-2bbda9e3c336/ovn-acl-logging/0.log"
May 26 11:25:01.907413 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:25:01.907260    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-js8cc" event={"ID":"7264c907-f536-4719-905a-2bbda9e3c336","Type":"ContainerStarted","Data":"0a27741c6a5c0cef50c8784496b97704b5287886c8310c9d8d473a1ffdf9e4d2"}
May 26 11:25:02.047653 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:25:02.047558    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2ce3eec7-5b62-4e4f-ad7f-ffc1eda0c2d1-original-pull-secret\") pod \"global-pull-secret-syncer-5jpd7\" (UID: \"2ce3eec7-5b62-4e4f-ad7f-ffc1eda0c2d1\") " pod="kube-system/global-pull-secret-syncer-5jpd7"
May 26 11:25:02.047820 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:25:02.047715    2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
May 26 11:25:02.047820 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:25:02.047786    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2ce3eec7-5b62-4e4f-ad7f-ffc1eda0c2d1-original-pull-secret podName:2ce3eec7-5b62-4e4f-ad7f-ffc1eda0c2d1 nodeName:}" failed. No retries permitted until 2026-05-26 11:25:18.047769435 +0000 UTC m=+40.949979753 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2ce3eec7-5b62-4e4f-ad7f-ffc1eda0c2d1-original-pull-secret") pod "global-pull-secret-syncer-5jpd7" (UID: "2ce3eec7-5b62-4e4f-ad7f-ffc1eda0c2d1") : object "kube-system"/"original-pull-secret" not registered
May 26 11:25:02.671348 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:25:02.671318    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k7jgt"
May 26 11:25:02.671514 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:25:02.671423    2572 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k7jgt" podUID="dede2966-ea61-4174-b667-c6fb46152d91"
May 26 11:25:02.746652 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:25:02.746622    2572 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-n4h7j"
May 26 11:25:02.747205 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:25:02.747168    2572 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-n4h7j"
May 26 11:25:02.765255 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:25:02.764703    2572 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6g66v" podStartSLOduration=4.562264497 podStartE2EDuration="24.76468718s" podCreationTimestamp="2026-05-26 11:24:38 +0000 UTC" firstStartedPulling="2026-05-26 11:24:40.191440652 +0000 UTC m=+3.093650974" lastFinishedPulling="2026-05-26 11:25:00.393863336 +0000 UTC m=+23.296073657" observedRunningTime="2026-05-26 11:25:00.925665265 +0000 UTC m=+23.827875596" watchObservedRunningTime="2026-05-26 11:25:02.76468718 +0000 UTC m=+25.666897510"
May 26 11:25:02.910459 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:25:02.910422    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vjbr5" event={"ID":"9e35a325-e1c3-4e2f-9ae5-b4120b6f1b60","Type":"ContainerStarted","Data":"4754b1a5f312d43bd7349c2db4c236e02870d08ee4e98aac21016d3d2de05f86"}
May 26 11:25:02.911159 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:25:02.910685    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-n4h7j"
May 26 11:25:02.911159 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:25:02.911122    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-n4h7j"
May 26 11:25:03.671222 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:25:03.671037    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cc5xb"
May 26 11:25:03.671397 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:25:03.671130    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5jpd7"
May 26 11:25:03.671397 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:25:03.671289    2572 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cc5xb" podUID="e87b53a4-1dca-46b1-bbd4-e484919093c4"
May 26 11:25:03.671474 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:25:03.671403    2572 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5jpd7" podUID="2ce3eec7-5b62-4e4f-ad7f-ffc1eda0c2d1"
May 26 11:25:03.914977 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:25:03.914947    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-js8cc_7264c907-f536-4719-905a-2bbda9e3c336/ovn-acl-logging/0.log"
May 26 11:25:03.915414 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:25:03.915306    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-js8cc" event={"ID":"7264c907-f536-4719-905a-2bbda9e3c336","Type":"ContainerStarted","Data":"c9668f24b6b4ea7b3129c9cad1efb1d18f4cd27ff187234ee6ca1bb02515756e"}
May 26 11:25:03.915640 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:25:03.915615    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-js8cc"
May 26 11:25:03.915791 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:25:03.915775    2572 scope.go:117] "RemoveContainer" containerID="c2466d41a89b7a36e4e710eb18868a895d6c1fadf830945e10645d9b81510111"
May 26 11:25:03.917087 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:25:03.917064    2572 generic.go:358] "Generic (PLEG): container finished" podID="9e35a325-e1c3-4e2f-9ae5-b4120b6f1b60" containerID="4754b1a5f312d43bd7349c2db4c236e02870d08ee4e98aac21016d3d2de05f86" exitCode=0
May 26 11:25:03.917180 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:25:03.917157    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vjbr5" event={"ID":"9e35a325-e1c3-4e2f-9ae5-b4120b6f1b60","Type":"ContainerDied","Data":"4754b1a5f312d43bd7349c2db4c236e02870d08ee4e98aac21016d3d2de05f86"}
May 26 11:25:03.933327 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:25:03.933303    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-js8cc"
May 26 11:25:04.670557 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:25:04.670526    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k7jgt"
May 26 11:25:04.670712 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:25:04.670668    2572 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k7jgt" podUID="dede2966-ea61-4174-b667-c6fb46152d91"
May 26 11:25:04.921431 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:25:04.921351    2572 generic.go:358] "Generic (PLEG): container finished" podID="9e35a325-e1c3-4e2f-9ae5-b4120b6f1b60" containerID="f1de2290185c645d36356f879b33f7d88235595e08113c16840f8adc8407ecfc" exitCode=0
May 26 11:25:04.921905 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:25:04.921449    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vjbr5" event={"ID":"9e35a325-e1c3-4e2f-9ae5-b4120b6f1b60","Type":"ContainerDied","Data":"f1de2290185c645d36356f879b33f7d88235595e08113c16840f8adc8407ecfc"}
May 26 11:25:04.925553 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:25:04.925539    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-js8cc_7264c907-f536-4719-905a-2bbda9e3c336/ovn-acl-logging/0.log"
May 26 11:25:04.925923 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:25:04.925904    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-js8cc" event={"ID":"7264c907-f536-4719-905a-2bbda9e3c336","Type":"ContainerStarted","Data":"a7c241a2452b3dacd0b81e8606ee2274cfb545931508f1f8bed7a17ea8f240b6"}
May 26 11:25:04.926416 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:25:04.926403    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-js8cc"
May 26 11:25:04.926499 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:25:04.926420    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-js8cc"
May 26 11:25:04.941699 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:25:04.941679    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-js8cc"
May 26 11:25:05.003693 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:25:05.003641    2572 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-js8cc" podStartSLOduration=9.083222456 podStartE2EDuration="27.003627127s" podCreationTimestamp="2026-05-26 11:24:38 +0000 UTC" firstStartedPulling="2026-05-26 11:24:40.16359742 +0000 UTC m=+3.065807736" lastFinishedPulling="2026-05-26 11:24:58.084002085 +0000 UTC m=+20.986212407" observedRunningTime="2026-05-26 11:25:05.001369038 +0000 UTC m=+27.903579369" watchObservedRunningTime="2026-05-26 11:25:05.003627127 +0000 UTC m=+27.905837456"
May 26 11:25:05.115840 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:25:05.115809    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-5jpd7"]
May 26 11:25:05.115988 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:25:05.115932    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5jpd7"
May 26 11:25:05.116023 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:25:05.116008    2572 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5jpd7" podUID="2ce3eec7-5b62-4e4f-ad7f-ffc1eda0c2d1"
May 26 11:25:05.118114 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:25:05.118091    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-k7jgt"]
May 26 11:25:05.118258 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:25:05.118180    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k7jgt"
May 26 11:25:05.118321 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:25:05.118302    2572 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k7jgt" podUID="dede2966-ea61-4174-b667-c6fb46152d91"
May 26 11:25:05.122314 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:25:05.122294    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-cc5xb"]
May 26 11:25:05.122415 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:25:05.122376    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cc5xb"
May 26 11:25:05.122472 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:25:05.122438    2572 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cc5xb" podUID="e87b53a4-1dca-46b1-bbd4-e484919093c4"
May 26 11:25:05.930154 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:25:05.929972    2572 generic.go:358] "Generic (PLEG): container finished" podID="9e35a325-e1c3-4e2f-9ae5-b4120b6f1b60" containerID="8e65e5be2d28329f1f1133817ba062a186591fa6c0375f6abd8f94e78c1948d5" exitCode=0
May 26 11:25:05.930154 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:25:05.930040    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vjbr5" event={"ID":"9e35a325-e1c3-4e2f-9ae5-b4120b6f1b60","Type":"ContainerDied","Data":"8e65e5be2d28329f1f1133817ba062a186591fa6c0375f6abd8f94e78c1948d5"}
May 26 11:25:06.671064 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:25:06.671023    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5jpd7"
May 26 11:25:06.671244 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:25:06.671075    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cc5xb"
May 26 11:25:06.671244 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:25:06.671156    2572 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5jpd7" podUID="2ce3eec7-5b62-4e4f-ad7f-ffc1eda0c2d1"
May 26 11:25:06.671244 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:25:06.671207    2572 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cc5xb" podUID="e87b53a4-1dca-46b1-bbd4-e484919093c4"
May 26 11:25:06.671244 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:25:06.671227    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k7jgt"
May 26 11:25:06.671452 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:25:06.671335    2572 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k7jgt" podUID="dede2966-ea61-4174-b667-c6fb46152d91"
May 26 11:25:08.671467 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:25:08.671429    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cc5xb"
May 26 11:25:08.671467 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:25:08.671456    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5jpd7"
May 26 11:25:08.671467 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:25:08.671437    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k7jgt"
May 26 11:25:08.672147 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:25:08.671549    2572 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5jpd7" podUID="2ce3eec7-5b62-4e4f-ad7f-ffc1eda0c2d1"
May 26 11:25:08.672147 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:25:08.671627    2572 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k7jgt" podUID="dede2966-ea61-4174-b667-c6fb46152d91"
May 26 11:25:08.672147 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:25:08.671666    2572 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cc5xb" podUID="e87b53a4-1dca-46b1-bbd4-e484919093c4"
May 26 11:25:10.670995 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:25:10.670963    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cc5xb"
May 26 11:25:10.670995 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:25:10.670987    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5jpd7"
May 26 11:25:10.671574 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:25:10.670976    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k7jgt"
May 26 11:25:10.671574 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:25:10.671084    2572 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cc5xb" podUID="e87b53a4-1dca-46b1-bbd4-e484919093c4"
May 26 11:25:10.671574 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:25:10.671176    2572 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5jpd7" podUID="2ce3eec7-5b62-4e4f-ad7f-ffc1eda0c2d1"
May 26 11:25:10.671574 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:25:10.671270    2572 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-k7jgt" podUID="dede2966-ea61-4174-b667-c6fb46152d91"
May 26 11:25:10.965615 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:25:10.965583    2572 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-136-194.ec2.internal" event="NodeReady"
May 26 11:25:10.965789 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:25:10.965752    2572 kubelet_node_status.go:550] "Fast updating node status as it just became ready"
May 26 11:25:11.008198 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:25:11.008151    2572 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5898c6cd7c-xxs9f"]
May 26 11:25:11.053718 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:25:11.053691    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5898c6cd7c-xxs9f"]
May 26 11:25:11.053718 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:25:11.053719    2572 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-wb9kk"]
May 26 11:25:11.053951 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:25:11.053864    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5898c6cd7c-xxs9f"
May 26 11:25:11.057560 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:25:11.057528    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
May 26 11:25:11.057837 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:25:11.057787    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
May 26 11:25:11.058027 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:25:11.057993    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
May 26 11:25:11.058621 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:25:11.058595    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-d4h9p\""
May 26 11:25:11.073442 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:25:11.073260    2572 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-m6p64"]
May 26 11:25:11.073572 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:25:11.073539    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-wb9kk"
May 26 11:25:11.075967 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:25:11.075913    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
May 26 11:25:11.076078 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:25:11.076006    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
May 26 11:25:11.076236 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:25:11.076216    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-t6qhf\""
May 26 11:25:11.087435 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:25:11.087417    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
May 26 11:25:11.097593 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:25:11.097565    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-wb9kk"]
May 26 11:25:11.097593 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:25:11.097596    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-m6p64"]
May 26 11:25:11.097746 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:25:11.097720    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-m6p64"
May 26 11:25:11.101454 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:25:11.101435    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
May 26 11:25:11.101817 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:25:11.101798    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
May 26 11:25:11.101900 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:25:11.101861    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
May 26 11:25:11.101950 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:25:11.101912    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-rwmqw\""
May 26 11:25:11.221465 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:25:11.221392    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b64f52e3-f082-4c86-a1fa-d3aa470c591f-cert\") pod \"ingress-canary-m6p64\" (UID: \"b64f52e3-f082-4c86-a1fa-d3aa470c591f\") " pod="openshift-ingress-canary/ingress-canary-m6p64"
May 26 11:25:11.221465 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:25:11.221428    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/be4abb11-f351-4413-8833-01c4774eb128-trusted-ca\") pod \"image-registry-5898c6cd7c-xxs9f\" (UID: \"be4abb11-f351-4413-8833-01c4774eb128\") " pod="openshift-image-registry/image-registry-5898c6cd7c-xxs9f"
May 26 11:25:11.221465 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:25:11.221451    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/be4abb11-f351-4413-8833-01c4774eb128-registry-tls\") pod \"image-registry-5898c6cd7c-xxs9f\" (UID: \"be4abb11-f351-4413-8833-01c4774eb128\") " pod="openshift-image-registry/image-registry-5898c6cd7c-xxs9f"
May 26 11:25:11.221657 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:25:11.221475    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/be4abb11-f351-4413-8833-01c4774eb128-bound-sa-token\") pod \"image-registry-5898c6cd7c-xxs9f\" (UID: \"be4abb11-f351-4413-8833-01c4774eb128\") " pod="openshift-image-registry/image-registry-5898c6cd7c-xxs9f"
May 26 11:25:11.221657 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:25:11.221512    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/be4abb11-f351-4413-8833-01c4774eb128-installation-pull-secrets\") pod \"image-registry-5898c6cd7c-xxs9f\" (UID: \"be4abb11-f351-4413-8833-01c4774eb128\") " pod="openshift-image-registry/image-registry-5898c6cd7c-xxs9f"
May 26 11:25:11.221657 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:25:11.221596    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gxc6\" (UniqueName: \"kubernetes.io/projected/b64f52e3-f082-4c86-a1fa-d3aa470c591f-kube-api-access-6gxc6\") pod \"ingress-canary-m6p64\" (UID: \"b64f52e3-f082-4c86-a1fa-d3aa470c591f\") " pod="openshift-ingress-canary/ingress-canary-m6p64"
May 26 11:25:11.221657 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:25:11.221617    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4deafc2e-1624-4384-b9b2-8a6c369c3308-config-volume\") pod \"dns-default-wb9kk\" (UID: \"4deafc2e-1624-4384-b9b2-8a6c369c3308\") " pod="openshift-dns/dns-default-wb9kk"
May 26 11:25:11.221657 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:25:11.221632    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/be4abb11-f351-4413-8833-01c4774eb128-ca-trust-extracted\") pod \"image-registry-5898c6cd7c-xxs9f\" (UID: \"be4abb11-f351-4413-8833-01c4774eb128\") " pod="openshift-image-registry/image-registry-5898c6cd7c-xxs9f"
May 26 11:25:11.221657 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:25:11.221649    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/be4abb11-f351-4413-8833-01c4774eb128-registry-certificates\") pod \"image-registry-5898c6cd7c-xxs9f\" (UID: \"be4abb11-f351-4413-8833-01c4774eb128\") " pod="openshift-image-registry/image-registry-5898c6cd7c-xxs9f"
May 26 11:25:11.221833 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:25:11.221670    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlkrt\" (UniqueName: \"kubernetes.io/projected/be4abb11-f351-4413-8833-01c4774eb128-kube-api-access-zlkrt\") pod \"image-registry-5898c6cd7c-xxs9f\" (UID: \"be4abb11-f351-4413-8833-01c4774eb128\") " pod="openshift-image-registry/image-registry-5898c6cd7c-xxs9f"
May 26 11:25:11.221833 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:25:11.221744    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4deafc2e-1624-4384-b9b2-8a6c369c3308-metrics-tls\") pod \"dns-default-wb9kk\" (UID: \"4deafc2e-1624-4384-b9b2-8a6c369c3308\") " pod="openshift-dns/dns-default-wb9kk"
May 26 11:25:11.221833 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:25:11.221807    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/be4abb11-f351-4413-8833-01c4774eb128-image-registry-private-configuration\") pod \"image-registry-5898c6cd7c-xxs9f\" (UID: \"be4abb11-f351-4413-8833-01c4774eb128\") " pod="openshift-image-registry/image-registry-5898c6cd7c-xxs9f"
May 26 11:25:11.221930 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:25:11.221836    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4deafc2e-1624-4384-b9b2-8a6c369c3308-tmp-dir\") pod \"dns-default-wb9kk\" (UID: \"4deafc2e-1624-4384-b9b2-8a6c369c3308\") " pod="openshift-dns/dns-default-wb9kk"
May 26 11:25:11.221930 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:25:11.221853    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ps9bx\" (UniqueName: \"kubernetes.io/projected/4deafc2e-1624-4384-b9b2-8a6c369c3308-kube-api-access-ps9bx\") pod \"dns-default-wb9kk\" (UID: \"4deafc2e-1624-4384-b9b2-8a6c369c3308\") " pod="openshift-dns/dns-default-wb9kk"
May 26 11:25:11.322761 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:25:11.322721    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/be4abb11-f351-4413-8833-01c4774eb128-image-registry-private-configuration\") pod \"image-registry-5898c6cd7c-xxs9f\" (UID: \"be4abb11-f351-4413-8833-01c4774eb128\") " pod="openshift-image-registry/image-registry-5898c6cd7c-xxs9f"
May 26 11:25:11.322761 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:25:11.322765    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4deafc2e-1624-4384-b9b2-8a6c369c3308-tmp-dir\") pod \"dns-default-wb9kk\" (UID: \"4deafc2e-1624-4384-b9b2-8a6c369c3308\") " pod="openshift-dns/dns-default-wb9kk"
May 26 11:25:11.322956 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:25:11.322782    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-ps9bx\" (UniqueName: \"kubernetes.io/projected/4deafc2e-1624-4384-b9b2-8a6c369c3308-kube-api-access-ps9bx\") pod \"dns-default-wb9kk\" (UID: \"4deafc2e-1624-4384-b9b2-8a6c369c3308\") " pod="openshift-dns/dns-default-wb9kk"
May 26 11:25:11.322956 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:25:11.322810    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b64f52e3-f082-4c86-a1fa-d3aa470c591f-cert\") pod \"ingress-canary-m6p64\" (UID: \"b64f52e3-f082-4c86-a1fa-d3aa470c591f\") " pod="openshift-ingress-canary/ingress-canary-m6p64"
May 26 11:25:11.322956 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:25:11.322827    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/be4abb11-f351-4413-8833-01c4774eb128-trusted-ca\") pod \"image-registry-5898c6cd7c-xxs9f\" (UID: \"be4abb11-f351-4413-8833-01c4774eb128\") " pod="openshift-image-registry/image-registry-5898c6cd7c-xxs9f"
May 26 11:25:11.322956 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:25:11.322847    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/be4abb11-f351-4413-8833-01c4774eb128-registry-tls\") pod \"image-registry-5898c6cd7c-xxs9f\" (UID: \"be4abb11-f351-4413-8833-01c4774eb128\") " pod="openshift-image-registry/image-registry-5898c6cd7c-xxs9f"
May 26 11:25:11.322956 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:25:11.322862    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/be4abb11-f351-4413-8833-01c4774eb128-bound-sa-token\") pod \"image-registry-5898c6cd7c-xxs9f\" (UID: \"be4abb11-f351-4413-8833-01c4774eb128\") " pod="openshift-image-registry/image-registry-5898c6cd7c-xxs9f"
May 26 11:25:11.322956 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:25:11.322880    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/be4abb11-f351-4413-8833-01c4774eb128-installation-pull-secrets\") pod \"image-registry-5898c6cd7c-xxs9f\" (UID: \"be4abb11-f351-4413-8833-01c4774eb128\") " pod="openshift-image-registry/image-registry-5898c6cd7c-xxs9f"
May 26 11:25:11.322956 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:25:11.322927    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-6gxc6\" (UniqueName: \"kubernetes.io/projected/b64f52e3-f082-4c86-a1fa-d3aa470c591f-kube-api-access-6gxc6\") pod \"ingress-canary-m6p64\" (UID: \"b64f52e3-f082-4c86-a1fa-d3aa470c591f\") " pod="openshift-ingress-canary/ingress-canary-m6p64"
May 26 11:25:11.322956 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:25:11.322936    2572 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 26 11:25:11.322956 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:25:11.322954    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4deafc2e-1624-4384-b9b2-8a6c369c3308-config-volume\") pod \"dns-default-wb9kk\" (UID: \"4deafc2e-1624-4384-b9b2-8a6c369c3308\") " pod="openshift-dns/dns-default-wb9kk"
May 26 11:25:11.323388 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:25:11.322978    2572 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 26 11:25:11.323388 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:25:11.322999    2572 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5898c6cd7c-xxs9f: secret "image-registry-tls" not found
May 26 11:25:11.323388 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:25:11.323010    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b64f52e3-f082-4c86-a1fa-d3aa470c591f-cert podName:b64f52e3-f082-4c86-a1fa-d3aa470c591f nodeName:}" failed. No retries permitted until 2026-05-26 11:25:11.822986137 +0000 UTC m=+34.725196462 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/b64f52e3-f082-4c86-a1fa-d3aa470c591f-cert") pod "ingress-canary-m6p64" (UID: "b64f52e3-f082-4c86-a1fa-d3aa470c591f") : secret "canary-serving-cert" not found
May 26 11:25:11.323388 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:25:11.323049    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/be4abb11-f351-4413-8833-01c4774eb128-registry-tls podName:be4abb11-f351-4413-8833-01c4774eb128 nodeName:}" failed. No retries permitted until 2026-05-26 11:25:11.823031876 +0000 UTC m=+34.725242196 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/be4abb11-f351-4413-8833-01c4774eb128-registry-tls") pod "image-registry-5898c6cd7c-xxs9f" (UID: "be4abb11-f351-4413-8833-01c4774eb128") : secret "image-registry-tls" not found
May 26 11:25:11.323388 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:25:11.322982    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/be4abb11-f351-4413-8833-01c4774eb128-ca-trust-extracted\") pod \"image-registry-5898c6cd7c-xxs9f\" (UID: \"be4abb11-f351-4413-8833-01c4774eb128\") " pod="openshift-image-registry/image-registry-5898c6cd7c-xxs9f"
May 26 11:25:11.323388 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:25:11.323064    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4deafc2e-1624-4384-b9b2-8a6c369c3308-tmp-dir\") pod \"dns-default-wb9kk\" (UID: \"4deafc2e-1624-4384-b9b2-8a6c369c3308\") " pod="openshift-dns/dns-default-wb9kk"
May 26 11:25:11.323388 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:25:11.323096    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/be4abb11-f351-4413-8833-01c4774eb128-registry-certificates\") pod \"image-registry-5898c6cd7c-xxs9f\" (UID: \"be4abb11-f351-4413-8833-01c4774eb128\") " pod="openshift-image-registry/image-registry-5898c6cd7c-xxs9f"
May 26 11:25:11.323388 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:25:11.323123    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-zlkrt\" (UniqueName: \"kubernetes.io/projected/be4abb11-f351-4413-8833-01c4774eb128-kube-api-access-zlkrt\") pod \"image-registry-5898c6cd7c-xxs9f\" (UID: \"be4abb11-f351-4413-8833-01c4774eb128\") " pod="openshift-image-registry/image-registry-5898c6cd7c-xxs9f"
May 26 11:25:11.323388 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:25:11.323157    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4deafc2e-1624-4384-b9b2-8a6c369c3308-metrics-tls\") pod \"dns-default-wb9kk\" (UID: \"4deafc2e-1624-4384-b9b2-8a6c369c3308\") " pod="openshift-dns/dns-default-wb9kk"
May 26 11:25:11.323388 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:25:11.323287    2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 26 11:25:11.323388 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:25:11.323322    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4deafc2e-1624-4384-b9b2-8a6c369c3308-metrics-tls podName:4deafc2e-1624-4384-b9b2-8a6c369c3308 nodeName:}" failed. No retries permitted until 2026-05-26 11:25:11.823310975 +0000 UTC m=+34.725521302 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4deafc2e-1624-4384-b9b2-8a6c369c3308-metrics-tls") pod "dns-default-wb9kk" (UID: "4deafc2e-1624-4384-b9b2-8a6c369c3308") : secret "dns-default-metrics-tls" not found
May 26 11:25:11.323934 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:25:11.323636    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/be4abb11-f351-4413-8833-01c4774eb128-ca-trust-extracted\") pod \"image-registry-5898c6cd7c-xxs9f\" (UID: \"be4abb11-f351-4413-8833-01c4774eb128\") " pod="openshift-image-registry/image-registry-5898c6cd7c-xxs9f"
May 26 11:25:11.323934 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:25:11.323749    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/be4abb11-f351-4413-8833-01c4774eb128-registry-certificates\") pod \"image-registry-5898c6cd7c-xxs9f\" (UID: \"be4abb11-f351-4413-8833-01c4774eb128\") " pod="openshift-image-registry/image-registry-5898c6cd7c-xxs9f"
May 26 11:25:11.323934 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:25:11.323842    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4deafc2e-1624-4384-b9b2-8a6c369c3308-config-volume\") pod \"dns-default-wb9kk\" (UID: \"4deafc2e-1624-4384-b9b2-8a6c369c3308\") " pod="openshift-dns/dns-default-wb9kk"
May 26 11:25:11.323934 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:25:11.323890    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/be4abb11-f351-4413-8833-01c4774eb128-trusted-ca\") pod \"image-registry-5898c6cd7c-xxs9f\" (UID: \"be4abb11-f351-4413-8833-01c4774eb128\") " pod="openshift-image-registry/image-registry-5898c6cd7c-xxs9f"
May 26 11:25:11.326858 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:25:11.326838    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/be4abb11-f351-4413-8833-01c4774eb128-installation-pull-secrets\") pod \"image-registry-5898c6cd7c-xxs9f\" (UID: \"be4abb11-f351-4413-8833-01c4774eb128\") " pod="openshift-image-registry/image-registry-5898c6cd7c-xxs9f"
May 26 11:25:11.326922 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:25:11.326857    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/be4abb11-f351-4413-8833-01c4774eb128-image-registry-private-configuration\") pod \"image-registry-5898c6cd7c-xxs9f\" (UID: \"be4abb11-f351-4413-8833-01c4774eb128\") " pod="openshift-image-registry/image-registry-5898c6cd7c-xxs9f"
May 26 11:25:11.363287 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:25:11.363268    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gxc6\" (UniqueName: \"kubernetes.io/projected/b64f52e3-f082-4c86-a1fa-d3aa470c591f-kube-api-access-6gxc6\") pod \"ingress-canary-m6p64\" (UID: \"b64f52e3-f082-4c86-a1fa-d3aa470c591f\") " pod="openshift-ingress-canary/ingress-canary-m6p64"
May 26 11:25:11.365386 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:25:11.365364    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ps9bx\" (UniqueName: \"kubernetes.io/projected/4deafc2e-1624-4384-b9b2-8a6c369c3308-kube-api-access-ps9bx\") pod \"dns-default-wb9kk\" (UID: \"4deafc2e-1624-4384-b9b2-8a6c369c3308\") " pod="openshift-dns/dns-default-wb9kk"
May 26 11:25:11.368155 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:25:11.368131    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlkrt\" (UniqueName: \"kubernetes.io/projected/be4abb11-f351-4413-8833-01c4774eb128-kube-api-access-zlkrt\") pod \"image-registry-5898c6cd7c-xxs9f\" (UID: \"be4abb11-f351-4413-8833-01c4774eb128\") " pod="openshift-image-registry/image-registry-5898c6cd7c-xxs9f"
May 26 11:25:11.370074 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:25:11.370051    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/be4abb11-f351-4413-8833-01c4774eb128-bound-sa-token\") pod \"image-registry-5898c6cd7c-xxs9f\" (UID: \"be4abb11-f351-4413-8833-01c4774eb128\") " pod="openshift-image-registry/image-registry-5898c6cd7c-xxs9f"
May 26 11:25:11.424404 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:25:11.424373    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dede2966-ea61-4174-b667-c6fb46152d91-metrics-certs\") pod \"network-metrics-daemon-k7jgt\" (UID: \"dede2966-ea61-4174-b667-c6fb46152d91\") " pod="openshift-multus/network-metrics-daemon-k7jgt"
May 26 11:25:11.424526 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:25:11.424471    2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 26 11:25:11.424526 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:25:11.424520    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dede2966-ea61-4174-b667-c6fb46152d91-metrics-certs podName:dede2966-ea61-4174-b667-c6fb46152d91 nodeName:}" failed. No retries permitted until 2026-05-26 11:25:43.424506873 +0000 UTC m=+66.326717180 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dede2966-ea61-4174-b667-c6fb46152d91-metrics-certs") pod "network-metrics-daemon-k7jgt" (UID: "dede2966-ea61-4174-b667-c6fb46152d91") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 26 11:25:11.525513 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:25:11.525275    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-7gbx6\" (UniqueName: \"kubernetes.io/projected/e87b53a4-1dca-46b1-bbd4-e484919093c4-kube-api-access-7gbx6\") pod \"network-check-target-cc5xb\" (UID: \"e87b53a4-1dca-46b1-bbd4-e484919093c4\") " pod="openshift-network-diagnostics/network-check-target-cc5xb"
May 26 11:25:11.525641 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:25:11.525422    2572 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 26 11:25:11.525641 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:25:11.525538    2572 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 26 11:25:11.525641 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:25:11.525548    2572 projected.go:194] Error preparing data for projected volume kube-api-access-7gbx6 for pod openshift-network-diagnostics/network-check-target-cc5xb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 26 11:25:11.525641 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:25:11.525595    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e87b53a4-1dca-46b1-bbd4-e484919093c4-kube-api-access-7gbx6 podName:e87b53a4-1dca-46b1-bbd4-e484919093c4 nodeName:}" failed. No retries permitted until 2026-05-26 11:25:43.525581568 +0000 UTC m=+66.427791876 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-7gbx6" (UniqueName: "kubernetes.io/projected/e87b53a4-1dca-46b1-bbd4-e484919093c4-kube-api-access-7gbx6") pod "network-check-target-cc5xb" (UID: "e87b53a4-1dca-46b1-bbd4-e484919093c4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 26 11:25:11.827321 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:25:11.827290    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/be4abb11-f351-4413-8833-01c4774eb128-registry-tls\") pod \"image-registry-5898c6cd7c-xxs9f\" (UID: \"be4abb11-f351-4413-8833-01c4774eb128\") " pod="openshift-image-registry/image-registry-5898c6cd7c-xxs9f"
May 26 11:25:11.827663 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:25:11.827357    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4deafc2e-1624-4384-b9b2-8a6c369c3308-metrics-tls\") pod \"dns-default-wb9kk\" (UID: \"4deafc2e-1624-4384-b9b2-8a6c369c3308\") " pod="openshift-dns/dns-default-wb9kk"
May 26 11:25:11.827663 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:25:11.827399    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b64f52e3-f082-4c86-a1fa-d3aa470c591f-cert\") pod \"ingress-canary-m6p64\" (UID: \"b64f52e3-f082-4c86-a1fa-d3aa470c591f\") " pod="openshift-ingress-canary/ingress-canary-m6p64"
May 26 11:25:11.827663 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:25:11.827437    2572 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 26 11:25:11.827663 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:25:11.827456    2572 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5898c6cd7c-xxs9f: secret "image-registry-tls" not found
May 26 11:25:11.827663 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:25:11.827472    2572 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 26 11:25:11.827663 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:25:11.827487    2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 26 11:25:11.827663 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:25:11.827533    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/be4abb11-f351-4413-8833-01c4774eb128-registry-tls podName:be4abb11-f351-4413-8833-01c4774eb128 nodeName:}" failed. No retries permitted until 2026-05-26 11:25:12.827506258 +0000 UTC m=+35.729716565 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/be4abb11-f351-4413-8833-01c4774eb128-registry-tls") pod "image-registry-5898c6cd7c-xxs9f" (UID: "be4abb11-f351-4413-8833-01c4774eb128") : secret "image-registry-tls" not found
May 26 11:25:11.827663 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:25:11.827553    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4deafc2e-1624-4384-b9b2-8a6c369c3308-metrics-tls podName:4deafc2e-1624-4384-b9b2-8a6c369c3308 nodeName:}" failed. No retries permitted until 2026-05-26 11:25:12.827542688 +0000 UTC m=+35.729753010 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4deafc2e-1624-4384-b9b2-8a6c369c3308-metrics-tls") pod "dns-default-wb9kk" (UID: "4deafc2e-1624-4384-b9b2-8a6c369c3308") : secret "dns-default-metrics-tls" not found
May 26 11:25:11.827663 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:25:11.827567    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b64f52e3-f082-4c86-a1fa-d3aa470c591f-cert podName:b64f52e3-f082-4c86-a1fa-d3aa470c591f nodeName:}" failed. No retries permitted until 2026-05-26 11:25:12.827559605 +0000 UTC m=+35.729769919 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/b64f52e3-f082-4c86-a1fa-d3aa470c591f-cert") pod "ingress-canary-m6p64" (UID: "b64f52e3-f082-4c86-a1fa-d3aa470c591f") : secret "canary-serving-cert" not found
May 26 11:25:11.943032 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:25:11.943004    2572 generic.go:358] "Generic (PLEG): container finished" podID="9e35a325-e1c3-4e2f-9ae5-b4120b6f1b60" containerID="b912d206ab145d6d082864b8f8830e907ae9f6b2f13f92f19e75d93e99e92bff" exitCode=0
May 26 11:25:11.943125 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:25:11.943069    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vjbr5" event={"ID":"9e35a325-e1c3-4e2f-9ae5-b4120b6f1b60","Type":"ContainerDied","Data":"b912d206ab145d6d082864b8f8830e907ae9f6b2f13f92f19e75d93e99e92bff"}
May 26 11:25:12.671294 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:25:12.671262    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5jpd7"
May 26 11:25:12.671469 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:25:12.671298    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cc5xb"
May 26 11:25:12.671469 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:25:12.671411    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k7jgt"
May 26 11:25:12.674791 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:25:12.674772    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
May 26 11:25:12.674891 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:25:12.674779    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
May 26 11:25:12.674935 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:25:12.674898    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-2gx5q\""
May 26 11:25:12.675975 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:25:12.675952    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
May 26 11:25:12.676072 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:25:12.676004    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
May 26 11:25:12.676072 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:25:12.676007    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-nf2lb\""
May 26 11:25:12.837290 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:25:12.837252    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4deafc2e-1624-4384-b9b2-8a6c369c3308-metrics-tls\") pod \"dns-default-wb9kk\" (UID: \"4deafc2e-1624-4384-b9b2-8a6c369c3308\") " pod="openshift-dns/dns-default-wb9kk"
May 26 11:25:12.837658 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:25:12.837341    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b64f52e3-f082-4c86-a1fa-d3aa470c591f-cert\") pod \"ingress-canary-m6p64\" (UID: \"b64f52e3-f082-4c86-a1fa-d3aa470c591f\") " pod="openshift-ingress-canary/ingress-canary-m6p64"
May 26 11:25:12.837658 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:25:12.837373    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/be4abb11-f351-4413-8833-01c4774eb128-registry-tls\") pod \"image-registry-5898c6cd7c-xxs9f\" (UID: \"be4abb11-f351-4413-8833-01c4774eb128\") " pod="openshift-image-registry/image-registry-5898c6cd7c-xxs9f"
May 26 11:25:12.837658 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:25:12.837411    2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 26 11:25:12.837658 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:25:12.837476    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4deafc2e-1624-4384-b9b2-8a6c369c3308-metrics-tls podName:4deafc2e-1624-4384-b9b2-8a6c369c3308 nodeName:}" failed. No retries permitted until 2026-05-26 11:25:14.837454315 +0000 UTC m=+37.739664637 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4deafc2e-1624-4384-b9b2-8a6c369c3308-metrics-tls") pod "dns-default-wb9kk" (UID: "4deafc2e-1624-4384-b9b2-8a6c369c3308") : secret "dns-default-metrics-tls" not found
May 26 11:25:12.837658 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:25:12.837494    2572 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 26 11:25:12.837658 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:25:12.837560    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b64f52e3-f082-4c86-a1fa-d3aa470c591f-cert podName:b64f52e3-f082-4c86-a1fa-d3aa470c591f nodeName:}" failed. No retries permitted until 2026-05-26 11:25:14.837547623 +0000 UTC m=+37.739757934 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/b64f52e3-f082-4c86-a1fa-d3aa470c591f-cert") pod "ingress-canary-m6p64" (UID: "b64f52e3-f082-4c86-a1fa-d3aa470c591f") : secret "canary-serving-cert" not found
May 26 11:25:12.837658 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:25:12.837498    2572 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 26 11:25:12.837658 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:25:12.837579    2572 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5898c6cd7c-xxs9f: secret "image-registry-tls" not found
May 26 11:25:12.837658 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:25:12.837606    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/be4abb11-f351-4413-8833-01c4774eb128-registry-tls podName:be4abb11-f351-4413-8833-01c4774eb128 nodeName:}" failed. No retries permitted until 2026-05-26 11:25:14.837600275 +0000 UTC m=+37.739810582 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/be4abb11-f351-4413-8833-01c4774eb128-registry-tls") pod "image-registry-5898c6cd7c-xxs9f" (UID: "be4abb11-f351-4413-8833-01c4774eb128") : secret "image-registry-tls" not found
May 26 11:25:12.947664 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:25:12.947570    2572 generic.go:358] "Generic (PLEG): container finished" podID="9e35a325-e1c3-4e2f-9ae5-b4120b6f1b60" containerID="041bf0e12e52dc904879494b8f5a6e00c6632d239c62e88a19f7a789ef25c147" exitCode=0
May 26 11:25:12.947664 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:25:12.947632    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vjbr5" event={"ID":"9e35a325-e1c3-4e2f-9ae5-b4120b6f1b60","Type":"ContainerDied","Data":"041bf0e12e52dc904879494b8f5a6e00c6632d239c62e88a19f7a789ef25c147"}
May 26 11:25:13.952691 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:25:13.952652    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vjbr5" event={"ID":"9e35a325-e1c3-4e2f-9ae5-b4120b6f1b60","Type":"ContainerStarted","Data":"5a8bcad719a1aee911c3d32f16e32e3199d5b17acde7645970a34246233f64f5"}
May 26 11:25:13.980124 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:25:13.980075    2572 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-vjbr5" podStartSLOduration=5.509967252 podStartE2EDuration="36.980060109s" podCreationTimestamp="2026-05-26 11:24:37 +0000 UTC" firstStartedPulling="2026-05-26 11:24:40.188619449 +0000 UTC m=+3.090829759" lastFinishedPulling="2026-05-26 11:25:11.658712298 +0000 UTC m=+34.560922616" observedRunningTime="2026-05-26 11:25:13.978485864 +0000 UTC m=+36.880696206" watchObservedRunningTime="2026-05-26 11:25:13.980060109 +0000 UTC m=+36.882270437"
May 26 11:25:14.853807 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:25:14.853775    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4deafc2e-1624-4384-b9b2-8a6c369c3308-metrics-tls\") pod \"dns-default-wb9kk\" (UID: \"4deafc2e-1624-4384-b9b2-8a6c369c3308\") " pod="openshift-dns/dns-default-wb9kk"
May 26 11:25:14.853947 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:25:14.853835    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b64f52e3-f082-4c86-a1fa-d3aa470c591f-cert\") pod \"ingress-canary-m6p64\" (UID: \"b64f52e3-f082-4c86-a1fa-d3aa470c591f\") " pod="openshift-ingress-canary/ingress-canary-m6p64"
May 26 11:25:14.853947 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:25:14.853859    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/be4abb11-f351-4413-8833-01c4774eb128-registry-tls\") pod \"image-registry-5898c6cd7c-xxs9f\" (UID: \"be4abb11-f351-4413-8833-01c4774eb128\") " pod="openshift-image-registry/image-registry-5898c6cd7c-xxs9f"
May 26 11:25:14.853947 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:25:14.853916    2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 26 11:25:14.854043 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:25:14.853972    2572 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 26 11:25:14.854043 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:25:14.853975    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4deafc2e-1624-4384-b9b2-8a6c369c3308-metrics-tls podName:4deafc2e-1624-4384-b9b2-8a6c369c3308 nodeName:}" failed. No retries permitted until 2026-05-26 11:25:18.853961554 +0000 UTC m=+41.756171863 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4deafc2e-1624-4384-b9b2-8a6c369c3308-metrics-tls") pod "dns-default-wb9kk" (UID: "4deafc2e-1624-4384-b9b2-8a6c369c3308") : secret "dns-default-metrics-tls" not found
May 26 11:25:14.854043 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:25:14.853974    2572 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 26 11:25:14.854179 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:25:14.854052    2572 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5898c6cd7c-xxs9f: secret "image-registry-tls" not found
May 26 11:25:14.854179 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:25:14.854031    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b64f52e3-f082-4c86-a1fa-d3aa470c591f-cert podName:b64f52e3-f082-4c86-a1fa-d3aa470c591f nodeName:}" failed. No retries permitted until 2026-05-26 11:25:18.854019984 +0000 UTC m=+41.756230291 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/b64f52e3-f082-4c86-a1fa-d3aa470c591f-cert") pod "ingress-canary-m6p64" (UID: "b64f52e3-f082-4c86-a1fa-d3aa470c591f") : secret "canary-serving-cert" not found
May 26 11:25:14.854179 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:25:14.854096    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/be4abb11-f351-4413-8833-01c4774eb128-registry-tls podName:be4abb11-f351-4413-8833-01c4774eb128 nodeName:}" failed. No retries permitted until 2026-05-26 11:25:18.854086285 +0000 UTC m=+41.756296595 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/be4abb11-f351-4413-8833-01c4774eb128-registry-tls") pod "image-registry-5898c6cd7c-xxs9f" (UID: "be4abb11-f351-4413-8833-01c4774eb128") : secret "image-registry-tls" not found
May 26 11:25:18.076162 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:25:18.076108    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2ce3eec7-5b62-4e4f-ad7f-ffc1eda0c2d1-original-pull-secret\") pod \"global-pull-secret-syncer-5jpd7\" (UID: \"2ce3eec7-5b62-4e4f-ad7f-ffc1eda0c2d1\") " pod="kube-system/global-pull-secret-syncer-5jpd7"
May 26 11:25:18.080052 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:25:18.080025    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2ce3eec7-5b62-4e4f-ad7f-ffc1eda0c2d1-original-pull-secret\") pod \"global-pull-secret-syncer-5jpd7\" (UID: \"2ce3eec7-5b62-4e4f-ad7f-ffc1eda0c2d1\") " pod="kube-system/global-pull-secret-syncer-5jpd7"
May 26 11:25:18.080818 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:25:18.080799    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5jpd7"
May 26 11:25:18.205128 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:25:18.205096    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-5jpd7"]
May 26 11:25:18.208208 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:25:18.208163    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ce3eec7_5b62_4e4f_ad7f_ffc1eda0c2d1.slice/crio-74fdb1bec035da09f821b2a2de30772108e5fac94944567b3cc31cb44d1e15e5 WatchSource:0}: Error finding container 74fdb1bec035da09f821b2a2de30772108e5fac94944567b3cc31cb44d1e15e5: Status 404 returned error can't find the container with id 74fdb1bec035da09f821b2a2de30772108e5fac94944567b3cc31cb44d1e15e5
May 26 11:25:18.881706 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:25:18.881669    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4deafc2e-1624-4384-b9b2-8a6c369c3308-metrics-tls\") pod \"dns-default-wb9kk\" (UID: \"4deafc2e-1624-4384-b9b2-8a6c369c3308\") " pod="openshift-dns/dns-default-wb9kk"
May 26 11:25:18.881906 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:25:18.881761    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b64f52e3-f082-4c86-a1fa-d3aa470c591f-cert\") pod \"ingress-canary-m6p64\" (UID: \"b64f52e3-f082-4c86-a1fa-d3aa470c591f\") " pod="openshift-ingress-canary/ingress-canary-m6p64"
May 26 11:25:18.881906 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:25:18.881799    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/be4abb11-f351-4413-8833-01c4774eb128-registry-tls\") pod \"image-registry-5898c6cd7c-xxs9f\" (UID: \"be4abb11-f351-4413-8833-01c4774eb128\") " pod="openshift-image-registry/image-registry-5898c6cd7c-xxs9f"
May 26 11:25:18.881906 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:25:18.881857    2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 26 11:25:18.882034 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:25:18.881914    2572 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 26 11:25:18.882034 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:25:18.881946    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4deafc2e-1624-4384-b9b2-8a6c369c3308-metrics-tls podName:4deafc2e-1624-4384-b9b2-8a6c369c3308 nodeName:}" failed. No retries permitted until 2026-05-26 11:25:26.881925205 +0000 UTC m=+49.784135526 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4deafc2e-1624-4384-b9b2-8a6c369c3308-metrics-tls") pod "dns-default-wb9kk" (UID: "4deafc2e-1624-4384-b9b2-8a6c369c3308") : secret "dns-default-metrics-tls" not found
May 26 11:25:18.882034 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:25:18.881918    2572 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 26 11:25:18.882034 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:25:18.881963    2572 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5898c6cd7c-xxs9f: secret "image-registry-tls" not found
May 26 11:25:18.882034 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:25:18.881966    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b64f52e3-f082-4c86-a1fa-d3aa470c591f-cert podName:b64f52e3-f082-4c86-a1fa-d3aa470c591f nodeName:}" failed. No retries permitted until 2026-05-26 11:25:26.881956677 +0000 UTC m=+49.784166989 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/b64f52e3-f082-4c86-a1fa-d3aa470c591f-cert") pod "ingress-canary-m6p64" (UID: "b64f52e3-f082-4c86-a1fa-d3aa470c591f") : secret "canary-serving-cert" not found
May 26 11:25:18.882034 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:25:18.882002    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/be4abb11-f351-4413-8833-01c4774eb128-registry-tls podName:be4abb11-f351-4413-8833-01c4774eb128 nodeName:}" failed. No retries permitted until 2026-05-26 11:25:26.881985301 +0000 UTC m=+49.784195619 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/be4abb11-f351-4413-8833-01c4774eb128-registry-tls") pod "image-registry-5898c6cd7c-xxs9f" (UID: "be4abb11-f351-4413-8833-01c4774eb128") : secret "image-registry-tls" not found
May 26 11:25:18.962751 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:25:18.962715    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-5jpd7" event={"ID":"2ce3eec7-5b62-4e4f-ad7f-ffc1eda0c2d1","Type":"ContainerStarted","Data":"74fdb1bec035da09f821b2a2de30772108e5fac94944567b3cc31cb44d1e15e5"}
May 26 11:25:21.970090 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:25:21.970051    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-5jpd7" event={"ID":"2ce3eec7-5b62-4e4f-ad7f-ffc1eda0c2d1","Type":"ContainerStarted","Data":"8dd5eeba0a0b10396f06640dec702fe96bc782ed19ed62c31d06999b2a6e6b5c"}
May 26 11:25:21.986596 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:25:21.986316    2572 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-5jpd7" podStartSLOduration=32.551473326 podStartE2EDuration="35.986299897s" podCreationTimestamp="2026-05-26 11:24:46 +0000 UTC" firstStartedPulling="2026-05-26 11:25:18.209983577 +0000 UTC m=+41.112193883" lastFinishedPulling="2026-05-26 11:25:21.644810147 +0000 UTC m=+44.547020454" observedRunningTime="2026-05-26 11:25:21.986022787 +0000 UTC m=+44.888233117" watchObservedRunningTime="2026-05-26 11:25:21.986299897 +0000 UTC m=+44.888510226"
May 26 11:25:26.947854 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:25:26.947813    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b64f52e3-f082-4c86-a1fa-d3aa470c591f-cert\") pod \"ingress-canary-m6p64\" (UID: \"b64f52e3-f082-4c86-a1fa-d3aa470c591f\") " pod="openshift-ingress-canary/ingress-canary-m6p64"
May 26 11:25:26.948402 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:25:26.947863    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/be4abb11-f351-4413-8833-01c4774eb128-registry-tls\") pod \"image-registry-5898c6cd7c-xxs9f\" (UID: \"be4abb11-f351-4413-8833-01c4774eb128\") " pod="openshift-image-registry/image-registry-5898c6cd7c-xxs9f"
May 26 11:25:26.948402 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:25:26.947912    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4deafc2e-1624-4384-b9b2-8a6c369c3308-metrics-tls\") pod \"dns-default-wb9kk\" (UID: \"4deafc2e-1624-4384-b9b2-8a6c369c3308\") " pod="openshift-dns/dns-default-wb9kk"
May 26 11:25:26.948402 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:25:26.947974    2572 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 26 11:25:26.948402 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:25:26.948036    2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 26 11:25:26.948402 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:25:26.948042    2572 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 26 11:25:26.948402 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:25:26.948059    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b64f52e3-f082-4c86-a1fa-d3aa470c591f-cert podName:b64f52e3-f082-4c86-a1fa-d3aa470c591f nodeName:}" failed. No retries permitted until 2026-05-26 11:25:42.948042424 +0000 UTC m=+65.850252734 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/b64f52e3-f082-4c86-a1fa-d3aa470c591f-cert") pod "ingress-canary-m6p64" (UID: "b64f52e3-f082-4c86-a1fa-d3aa470c591f") : secret "canary-serving-cert" not found
May 26 11:25:26.948402 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:25:26.948061    2572 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5898c6cd7c-xxs9f: secret "image-registry-tls" not found
May 26 11:25:26.948402 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:25:26.948098    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4deafc2e-1624-4384-b9b2-8a6c369c3308-metrics-tls podName:4deafc2e-1624-4384-b9b2-8a6c369c3308 nodeName:}" failed. No retries permitted until 2026-05-26 11:25:42.948081036 +0000 UTC m=+65.850291355 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4deafc2e-1624-4384-b9b2-8a6c369c3308-metrics-tls") pod "dns-default-wb9kk" (UID: "4deafc2e-1624-4384-b9b2-8a6c369c3308") : secret "dns-default-metrics-tls" not found
May 26 11:25:26.948402 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:25:26.948121    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/be4abb11-f351-4413-8833-01c4774eb128-registry-tls podName:be4abb11-f351-4413-8833-01c4774eb128 nodeName:}" failed. No retries permitted until 2026-05-26 11:25:42.948113187 +0000 UTC m=+65.850323494 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/be4abb11-f351-4413-8833-01c4774eb128-registry-tls") pod "image-registry-5898c6cd7c-xxs9f" (UID: "be4abb11-f351-4413-8833-01c4774eb128") : secret "image-registry-tls" not found
May 26 11:25:36.942607 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:25:36.942578    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-js8cc"
May 26 11:25:42.956756 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:25:42.956717    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/be4abb11-f351-4413-8833-01c4774eb128-registry-tls\") pod \"image-registry-5898c6cd7c-xxs9f\" (UID: \"be4abb11-f351-4413-8833-01c4774eb128\") " pod="openshift-image-registry/image-registry-5898c6cd7c-xxs9f"
May 26 11:25:42.957278 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:25:42.956772    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4deafc2e-1624-4384-b9b2-8a6c369c3308-metrics-tls\") pod \"dns-default-wb9kk\" (UID: \"4deafc2e-1624-4384-b9b2-8a6c369c3308\") " pod="openshift-dns/dns-default-wb9kk"
May 26 11:25:42.957278 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:25:42.956814    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b64f52e3-f082-4c86-a1fa-d3aa470c591f-cert\") pod \"ingress-canary-m6p64\" (UID: \"b64f52e3-f082-4c86-a1fa-d3aa470c591f\") " pod="openshift-ingress-canary/ingress-canary-m6p64"
May 26 11:25:42.957278 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:25:42.956866    2572 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 26 11:25:42.957278 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:25:42.956885    2572 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5898c6cd7c-xxs9f: secret "image-registry-tls" not found
May 26 11:25:42.957278 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:25:42.956888    2572 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 26 11:25:42.957278 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:25:42.956937    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b64f52e3-f082-4c86-a1fa-d3aa470c591f-cert podName:b64f52e3-f082-4c86-a1fa-d3aa470c591f nodeName:}" failed. No retries permitted until 2026-05-26 11:26:14.956923012 +0000 UTC m=+97.859133318 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/b64f52e3-f082-4c86-a1fa-d3aa470c591f-cert") pod "ingress-canary-m6p64" (UID: "b64f52e3-f082-4c86-a1fa-d3aa470c591f") : secret "canary-serving-cert" not found
May 26 11:25:42.957278 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:25:42.956951    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/be4abb11-f351-4413-8833-01c4774eb128-registry-tls podName:be4abb11-f351-4413-8833-01c4774eb128 nodeName:}" failed. No retries permitted until 2026-05-26 11:26:14.956944896 +0000 UTC m=+97.859155203 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/be4abb11-f351-4413-8833-01c4774eb128-registry-tls") pod "image-registry-5898c6cd7c-xxs9f" (UID: "be4abb11-f351-4413-8833-01c4774eb128") : secret "image-registry-tls" not found
May 26 11:25:42.957278 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:25:42.956949    2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 26 11:25:42.957278 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:25:42.957026    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4deafc2e-1624-4384-b9b2-8a6c369c3308-metrics-tls podName:4deafc2e-1624-4384-b9b2-8a6c369c3308 nodeName:}" failed. No retries permitted until 2026-05-26 11:26:14.957005971 +0000 UTC m=+97.859216291 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4deafc2e-1624-4384-b9b2-8a6c369c3308-metrics-tls") pod "dns-default-wb9kk" (UID: "4deafc2e-1624-4384-b9b2-8a6c369c3308") : secret "dns-default-metrics-tls" not found
May 26 11:25:43.460166 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:25:43.460135    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dede2966-ea61-4174-b667-c6fb46152d91-metrics-certs\") pod \"network-metrics-daemon-k7jgt\" (UID: \"dede2966-ea61-4174-b667-c6fb46152d91\") " pod="openshift-multus/network-metrics-daemon-k7jgt"
May 26 11:25:43.462780 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:25:43.462760    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
May 26 11:25:43.471009 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:25:43.470992    2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
May 26 11:25:43.471100 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:25:43.471056    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dede2966-ea61-4174-b667-c6fb46152d91-metrics-certs podName:dede2966-ea61-4174-b667-c6fb46152d91 nodeName:}" failed. No retries permitted until 2026-05-26 11:26:47.471035758 +0000 UTC m=+130.373246081 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dede2966-ea61-4174-b667-c6fb46152d91-metrics-certs") pod "network-metrics-daemon-k7jgt" (UID: "dede2966-ea61-4174-b667-c6fb46152d91") : secret "metrics-daemon-secret" not found
May 26 11:25:43.560787 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:25:43.560760    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-7gbx6\" (UniqueName: \"kubernetes.io/projected/e87b53a4-1dca-46b1-bbd4-e484919093c4-kube-api-access-7gbx6\") pod \"network-check-target-cc5xb\" (UID: \"e87b53a4-1dca-46b1-bbd4-e484919093c4\") " pod="openshift-network-diagnostics/network-check-target-cc5xb"
May 26 11:25:43.563500 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:25:43.563482    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
May 26 11:25:43.573983 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:25:43.573962    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
May 26 11:25:43.584155 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:25:43.584125    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gbx6\" (UniqueName: \"kubernetes.io/projected/e87b53a4-1dca-46b1-bbd4-e484919093c4-kube-api-access-7gbx6\") pod \"network-check-target-cc5xb\" (UID: \"e87b53a4-1dca-46b1-bbd4-e484919093c4\") " pod="openshift-network-diagnostics/network-check-target-cc5xb"
May 26 11:25:43.593288 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:25:43.593271    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-nf2lb\""
May 26 11:25:43.601570 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:25:43.601557    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cc5xb"
May 26 11:25:43.729606 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:25:43.729404    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-cc5xb"]
May 26 11:25:43.731970 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:25:43.731943    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode87b53a4_1dca_46b1_bbd4_e484919093c4.slice/crio-34745abf8f6d0aacfb2413afda33a86d6bce8aa31e80deff9029038f16a434cd WatchSource:0}: Error finding container 34745abf8f6d0aacfb2413afda33a86d6bce8aa31e80deff9029038f16a434cd: Status 404 returned error can't find the container with id 34745abf8f6d0aacfb2413afda33a86d6bce8aa31e80deff9029038f16a434cd
May 26 11:25:44.017055 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:25:44.016975    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-cc5xb" event={"ID":"e87b53a4-1dca-46b1-bbd4-e484919093c4","Type":"ContainerStarted","Data":"34745abf8f6d0aacfb2413afda33a86d6bce8aa31e80deff9029038f16a434cd"}
May 26 11:25:47.024075 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:25:47.024036    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-cc5xb" event={"ID":"e87b53a4-1dca-46b1-bbd4-e484919093c4","Type":"ContainerStarted","Data":"7ae44b85ecf7bb6c4d2e13fade1b382ca520275a1aff6957cc206feb3d69dd53"}
May 26 11:25:47.024553 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:25:47.024153    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-cc5xb"
May 26 11:25:47.041866 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:25:47.041821    2572 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-cc5xb" podStartSLOduration=67.56559407 podStartE2EDuration="1m10.041808806s" podCreationTimestamp="2026-05-26 11:24:37 +0000 UTC" firstStartedPulling="2026-05-26 11:25:43.733810232 +0000 UTC m=+66.636020539" lastFinishedPulling="2026-05-26 11:25:46.210024956 +0000 UTC m=+69.112235275" observedRunningTime="2026-05-26 11:25:47.041112115 +0000 UTC m=+69.943322445" watchObservedRunningTime="2026-05-26 11:25:47.041808806 +0000 UTC m=+69.944019134"
May 26 11:26:14.986332 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:26:14.986244    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/be4abb11-f351-4413-8833-01c4774eb128-registry-tls\") pod \"image-registry-5898c6cd7c-xxs9f\" (UID: \"be4abb11-f351-4413-8833-01c4774eb128\") " pod="openshift-image-registry/image-registry-5898c6cd7c-xxs9f"
May 26 11:26:14.986332 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:26:14.986311    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4deafc2e-1624-4384-b9b2-8a6c369c3308-metrics-tls\") pod \"dns-default-wb9kk\" (UID: \"4deafc2e-1624-4384-b9b2-8a6c369c3308\") " pod="openshift-dns/dns-default-wb9kk"
May 26 11:26:14.986790 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:26:14.986350    2572 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 26 11:26:14.986790 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:26:14.986361    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b64f52e3-f082-4c86-a1fa-d3aa470c591f-cert\") pod \"ingress-canary-m6p64\" (UID: \"b64f52e3-f082-4c86-a1fa-d3aa470c591f\") " pod="openshift-ingress-canary/ingress-canary-m6p64"
May 26 11:26:14.986790 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:26:14.986368    2572 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5898c6cd7c-xxs9f: secret "image-registry-tls" not found
May 26 11:26:14.986790 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:26:14.986433    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/be4abb11-f351-4413-8833-01c4774eb128-registry-tls podName:be4abb11-f351-4413-8833-01c4774eb128 nodeName:}" failed. No retries permitted until 2026-05-26 11:27:18.986411428 +0000 UTC m=+161.888621743 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/be4abb11-f351-4413-8833-01c4774eb128-registry-tls") pod "image-registry-5898c6cd7c-xxs9f" (UID: "be4abb11-f351-4413-8833-01c4774eb128") : secret "image-registry-tls" not found
May 26 11:26:14.986790 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:26:14.986441    2572 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 26 11:26:14.986790 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:26:14.986475    2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 26 11:26:14.986790 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:26:14.986750    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b64f52e3-f082-4c86-a1fa-d3aa470c591f-cert podName:b64f52e3-f082-4c86-a1fa-d3aa470c591f nodeName:}" failed. No retries permitted until 2026-05-26 11:27:18.986474447 +0000 UTC m=+161.888684768 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/b64f52e3-f082-4c86-a1fa-d3aa470c591f-cert") pod "ingress-canary-m6p64" (UID: "b64f52e3-f082-4c86-a1fa-d3aa470c591f") : secret "canary-serving-cert" not found
May 26 11:26:14.987117 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:26:14.986876    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4deafc2e-1624-4384-b9b2-8a6c369c3308-metrics-tls podName:4deafc2e-1624-4384-b9b2-8a6c369c3308 nodeName:}" failed. No retries permitted until 2026-05-26 11:27:18.986862094 +0000 UTC m=+161.889072401 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4deafc2e-1624-4384-b9b2-8a6c369c3308-metrics-tls") pod "dns-default-wb9kk" (UID: "4deafc2e-1624-4384-b9b2-8a6c369c3308") : secret "dns-default-metrics-tls" not found
May 26 11:26:18.028674 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:26:18.028645    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-cc5xb"
May 26 11:26:47.498710 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:26:47.498674    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dede2966-ea61-4174-b667-c6fb46152d91-metrics-certs\") pod \"network-metrics-daemon-k7jgt\" (UID: \"dede2966-ea61-4174-b667-c6fb46152d91\") " pod="openshift-multus/network-metrics-daemon-k7jgt"
May 26 11:26:47.499170 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:26:47.498830    2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
May 26 11:26:47.499170 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:26:47.498906    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dede2966-ea61-4174-b667-c6fb46152d91-metrics-certs podName:dede2966-ea61-4174-b667-c6fb46152d91 nodeName:}" failed. No retries permitted until 2026-05-26 11:28:49.498889511 +0000 UTC m=+252.401099824 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dede2966-ea61-4174-b667-c6fb46152d91-metrics-certs") pod "network-metrics-daemon-k7jgt" (UID: "dede2966-ea61-4174-b667-c6fb46152d91") : secret "metrics-daemon-secret" not found
May 26 11:27:14.081020 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:27:14.080978    2572 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-5898c6cd7c-xxs9f" podUID="be4abb11-f351-4413-8833-01c4774eb128"
May 26 11:27:14.087164 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:27:14.087137    2572 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-wb9kk" podUID="4deafc2e-1624-4384-b9b2-8a6c369c3308"
May 26 11:27:14.107264 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:27:14.107245    2572 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-ingress-canary/ingress-canary-m6p64" podUID="b64f52e3-f082-4c86-a1fa-d3aa470c591f"
May 26 11:27:14.182866 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:14.182841    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-wb9kk"
May 26 11:27:15.686372 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:27:15.686337    2572 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-k7jgt" podUID="dede2966-ea61-4174-b667-c6fb46152d91"
May 26 11:27:19.007748 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:19.007697    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b64f52e3-f082-4c86-a1fa-d3aa470c591f-cert\") pod \"ingress-canary-m6p64\" (UID: \"b64f52e3-f082-4c86-a1fa-d3aa470c591f\") " pod="openshift-ingress-canary/ingress-canary-m6p64"
May 26 11:27:19.007748 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:19.007746    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/be4abb11-f351-4413-8833-01c4774eb128-registry-tls\") pod \"image-registry-5898c6cd7c-xxs9f\" (UID: \"be4abb11-f351-4413-8833-01c4774eb128\") " pod="openshift-image-registry/image-registry-5898c6cd7c-xxs9f"
May 26 11:27:19.008246 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:19.007781    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4deafc2e-1624-4384-b9b2-8a6c369c3308-metrics-tls\") pod \"dns-default-wb9kk\" (UID: \"4deafc2e-1624-4384-b9b2-8a6c369c3308\") " pod="openshift-dns/dns-default-wb9kk"
May 26 11:27:19.008246 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:27:19.007844    2572 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 26 11:27:19.008246 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:27:19.007920    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b64f52e3-f082-4c86-a1fa-d3aa470c591f-cert podName:b64f52e3-f082-4c86-a1fa-d3aa470c591f nodeName:}" failed. No retries permitted until 2026-05-26 11:29:21.007905079 +0000 UTC m=+283.910115385 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/b64f52e3-f082-4c86-a1fa-d3aa470c591f-cert") pod "ingress-canary-m6p64" (UID: "b64f52e3-f082-4c86-a1fa-d3aa470c591f") : secret "canary-serving-cert" not found
May 26 11:27:19.008246 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:27:19.007858    2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 26 11:27:19.008246 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:27:19.007927    2572 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 26 11:27:19.008246 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:27:19.007950    2572 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5898c6cd7c-xxs9f: secret "image-registry-tls" not found
May 26 11:27:19.008246 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:27:19.007967    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4deafc2e-1624-4384-b9b2-8a6c369c3308-metrics-tls podName:4deafc2e-1624-4384-b9b2-8a6c369c3308 nodeName:}" failed. No retries permitted until 2026-05-26 11:29:21.007955832 +0000 UTC m=+283.910166139 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4deafc2e-1624-4384-b9b2-8a6c369c3308-metrics-tls") pod "dns-default-wb9kk" (UID: "4deafc2e-1624-4384-b9b2-8a6c369c3308") : secret "dns-default-metrics-tls" not found
May 26 11:27:19.008246 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:27:19.007997    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/be4abb11-f351-4413-8833-01c4774eb128-registry-tls podName:be4abb11-f351-4413-8833-01c4774eb128 nodeName:}" failed. No retries permitted until 2026-05-26 11:29:21.007982942 +0000 UTC m=+283.910193250 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/be4abb11-f351-4413-8833-01c4774eb128-registry-tls") pod "image-registry-5898c6cd7c-xxs9f" (UID: "be4abb11-f351-4413-8833-01c4774eb128") : secret "image-registry-tls" not found
May 26 11:27:20.858221 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:20.858171    2572 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-697587599-hls8q"]
May 26 11:27:20.860980 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:20.860965    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-hls8q"
May 26 11:27:20.863998 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:20.863978    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
May 26 11:27:20.865038 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:20.865021    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
May 26 11:27:20.865140 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:20.865030    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
May 26 11:27:20.865140 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:20.865073    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-cfdn9\""
May 26 11:27:20.878508 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:20.878484    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-697587599-hls8q"]
May 26 11:27:20.883695 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:20.883671    2572 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5cf8c89d64-vsvjj"]
May 26 11:27:20.886437 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:20.886421    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5cf8c89d64-vsvjj"
May 26 11:27:20.890401 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:20.890382    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
May 26 11:27:20.890948 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:20.890933    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
May 26 11:27:20.891385 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:20.891373    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-z2nbm\""
May 26 11:27:20.897786 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:20.897763    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5cf8c89d64-vsvjj"]
May 26 11:27:20.919220 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:20.919181    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ed80247a-4151-4422-b05e-a1764ec5d650-samples-operator-tls\") pod \"cluster-samples-operator-697587599-hls8q\" (UID: \"ed80247a-4151-4422-b05e-a1764ec5d650\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-hls8q"
May 26 11:27:20.919334 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:20.919289    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ttxp\" (UniqueName: \"kubernetes.io/projected/ed80247a-4151-4422-b05e-a1764ec5d650-kube-api-access-4ttxp\") pod \"cluster-samples-operator-697587599-hls8q\" (UID: \"ed80247a-4151-4422-b05e-a1764ec5d650\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-hls8q"
May 26 11:27:20.974799 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:20.974762    2572 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6c75959455-xqrk8"]
May 26 11:27:20.977597 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:20.977579    2572 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-76dff48d59-5bfgg"]
May 26 11:27:20.977743 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:20.977723    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-xqrk8"
May 26 11:27:20.980563 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:20.980543    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-76dff48d59-5bfgg"
May 26 11:27:20.982117 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:20.982094    2572 status_manager.go:895] "Failed to get status for pod" podUID="97673f35-daf2-4eaf-b6cc-a119472f65c3" pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-xqrk8" err="pods \"cluster-monitoring-operator-6c75959455-xqrk8\" is forbidden: User \"system:node:ip-10-0-136-194.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-monitoring\": no relationship found between node 'ip-10-0-136-194.ec2.internal' and this object"
May 26 11:27:20.982311 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:27:20.982293    2572 reflector.go:200] "Failed to watch" err="failed to list *v1.Secret: secrets \"cluster-monitoring-operator-tls\" is forbidden: User \"system:node:ip-10-0-136-194.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-monitoring\": no relationship found between node 'ip-10-0-136-194.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" type="*v1.Secret"
May 26 11:27:20.982442 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:27:20.982427    2572 reflector.go:200] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"openshift-service-ca.crt\" is forbidden: User \"system:node:ip-10-0-136-194.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-monitoring\": no relationship found between node 'ip-10-0-136-194.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap"
May 26 11:27:20.983004 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:27:20.982978    2572 reflector.go:200] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"telemetry-config\" is forbidden: User \"system:node:ip-10-0-136-194.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-monitoring\": no relationship found between node 'ip-10-0-136-194.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" type="*v1.ConfigMap"
May 26 11:27:20.983109 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:20.983087    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
May 26 11:27:20.983638 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:20.983625    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-g8gk4\""
May 26 11:27:20.985125 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:20.985107    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
May 26 11:27:20.986000 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:20.985986    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
May 26 11:27:20.986584 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:20.986571    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
May 26 11:27:20.986808 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:20.986796    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
May 26 11:27:20.986863 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:20.986837    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-p4nx6\""
May 26 11:27:21.020220 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:21.020182    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6c75959455-xqrk8"]
May 26 11:27:21.020390 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:21.020369    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ed80247a-4151-4422-b05e-a1764ec5d650-samples-operator-tls\") pod \"cluster-samples-operator-697587599-hls8q\" (UID: \"ed80247a-4151-4422-b05e-a1764ec5d650\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-hls8q"
May 26 11:27:21.020462 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:21.020408    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gd6zq\" (UniqueName: \"kubernetes.io/projected/97673f35-daf2-4eaf-b6cc-a119472f65c3-kube-api-access-gd6zq\") pod \"cluster-monitoring-operator-6c75959455-xqrk8\" (UID: \"97673f35-daf2-4eaf-b6cc-a119472f65c3\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-xqrk8"
May 26 11:27:21.020522 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:21.020472    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-4ttxp\" (UniqueName: \"kubernetes.io/projected/ed80247a-4151-4422-b05e-a1764ec5d650-kube-api-access-4ttxp\") pod \"cluster-samples-operator-697587599-hls8q\" (UID: \"ed80247a-4151-4422-b05e-a1764ec5d650\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-hls8q"
May 26 11:27:21.020522 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:21.020500    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c6373342-dd07-4497-a4f3-978ea9613b6e-serving-cert\") pod \"console-operator-76dff48d59-5bfgg\" (UID: \"c6373342-dd07-4497-a4f3-978ea9613b6e\") " pod="openshift-console-operator/console-operator-76dff48d59-5bfgg"
May 26 11:27:21.020522 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:27:21.020520    2572 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
May 26 11:27:21.020664 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:27:21.020565    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed80247a-4151-4422-b05e-a1764ec5d650-samples-operator-tls podName:ed80247a-4151-4422-b05e-a1764ec5d650 nodeName:}" failed. No retries permitted until 2026-05-26 11:27:21.520552554 +0000 UTC m=+164.422762865 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/ed80247a-4151-4422-b05e-a1764ec5d650-samples-operator-tls") pod "cluster-samples-operator-697587599-hls8q" (UID: "ed80247a-4151-4422-b05e-a1764ec5d650") : secret "samples-operator-tls" not found
May 26 11:27:21.020664 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:21.020599    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c6373342-dd07-4497-a4f3-978ea9613b6e-trusted-ca\") pod \"console-operator-76dff48d59-5bfgg\" (UID: \"c6373342-dd07-4497-a4f3-978ea9613b6e\") " pod="openshift-console-operator/console-operator-76dff48d59-5bfgg"
May 26 11:27:21.020664 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:21.020625    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bn5m\" (UniqueName: \"kubernetes.io/projected/c6373342-dd07-4497-a4f3-978ea9613b6e-kube-api-access-7bn5m\") pod \"console-operator-76dff48d59-5bfgg\" (UID: \"c6373342-dd07-4497-a4f3-978ea9613b6e\") " pod="openshift-console-operator/console-operator-76dff48d59-5bfgg"
May 26 11:27:21.020807 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:21.020667    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xpkqm\" (UniqueName: \"kubernetes.io/projected/6aaefb25-f45b-44c1-8576-f0e9f4dd5977-kube-api-access-xpkqm\") pod \"volume-data-source-validator-5cf8c89d64-vsvjj\" (UID: \"6aaefb25-f45b-44c1-8576-f0e9f4dd5977\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5cf8c89d64-vsvjj"
May 26 11:27:21.020807 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:21.020718    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/97673f35-daf2-4eaf-b6cc-a119472f65c3-telemetry-config\") pod \"cluster-monitoring-operator-6c75959455-xqrk8\" (UID: \"97673f35-daf2-4eaf-b6cc-a119472f65c3\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-xqrk8"
May 26 11:27:21.020807 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:21.020767    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c6373342-dd07-4497-a4f3-978ea9613b6e-config\") pod \"console-operator-76dff48d59-5bfgg\" (UID: \"c6373342-dd07-4497-a4f3-978ea9613b6e\") " pod="openshift-console-operator/console-operator-76dff48d59-5bfgg"
May 26 11:27:21.020807 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:21.020797    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/97673f35-daf2-4eaf-b6cc-a119472f65c3-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6c75959455-xqrk8\" (UID: \"97673f35-daf2-4eaf-b6cc-a119472f65c3\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-xqrk8"
May 26 11:27:21.024081 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:21.024058    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-76dff48d59-5bfgg"]
May 26 11:27:21.024566 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:21.024548    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
May 26 11:27:21.052363 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:21.052339    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ttxp\" (UniqueName: \"kubernetes.io/projected/ed80247a-4151-4422-b05e-a1764ec5d650-kube-api-access-4ttxp\") pod \"cluster-samples-operator-697587599-hls8q\" (UID: \"ed80247a-4151-4422-b05e-a1764ec5d650\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-hls8q"
May 26 11:27:21.088652 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:21.088631    2572 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-57b5b68b8-sfz45"]
May 26 11:27:21.091483 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:21.091469    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-sfz45"
May 26 11:27:21.094243 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:21.094221    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
May 26 11:27:21.094364 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:21.094344    2572 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-7f44b6dd87-dpzf5"]
May 26 11:27:21.095328 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:21.095302    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
May 26 11:27:21.095439 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:21.095344    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
May 26 11:27:21.095439 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:21.095302    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-9vt7w\""
May 26 11:27:21.095439 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:21.095309    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
May 26 11:27:21.097250 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:21.097233    2572 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-ds944"]
May 26 11:27:21.097391 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:21.097369    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7f44b6dd87-dpzf5"
May 26 11:27:21.099965 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:21.099947    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-ds944"
May 26 11:27:21.100735 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:21.100717    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-phkbw\""
May 26 11:27:21.103625 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:21.103609    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
May 26 11:27:21.103883 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:21.103870    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
May 26 11:27:21.103965 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:21.103892    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
May 26 11:27:21.103965 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:21.103918    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
May 26 11:27:21.104133 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:21.104116    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-cqktd\""
May 26 11:27:21.121463 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:21.121444    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c6373342-dd07-4497-a4f3-978ea9613b6e-config\") pod \"console-operator-76dff48d59-5bfgg\" (UID: \"c6373342-dd07-4497-a4f3-978ea9613b6e\") " pod="openshift-console-operator/console-operator-76dff48d59-5bfgg"
May 26 11:27:21.121537 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:21.121477    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/97673f35-daf2-4eaf-b6cc-a119472f65c3-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6c75959455-xqrk8\" (UID: \"97673f35-daf2-4eaf-b6cc-a119472f65c3\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-xqrk8"
May 26 11:27:21.121537 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:21.121513    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-gd6zq\" (UniqueName: \"kubernetes.io/projected/97673f35-daf2-4eaf-b6cc-a119472f65c3-kube-api-access-gd6zq\") pod \"cluster-monitoring-operator-6c75959455-xqrk8\" (UID: \"97673f35-daf2-4eaf-b6cc-a119472f65c3\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-xqrk8"
May 26 11:27:21.121626 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:21.121567    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c6373342-dd07-4497-a4f3-978ea9613b6e-serving-cert\") pod \"console-operator-76dff48d59-5bfgg\" (UID: \"c6373342-dd07-4497-a4f3-978ea9613b6e\") " pod="openshift-console-operator/console-operator-76dff48d59-5bfgg"
May 26 11:27:21.121626 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:21.121614    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c6373342-dd07-4497-a4f3-978ea9613b6e-trusted-ca\") pod \"console-operator-76dff48d59-5bfgg\" (UID: \"c6373342-dd07-4497-a4f3-978ea9613b6e\") " pod="openshift-console-operator/console-operator-76dff48d59-5bfgg"
May 26 11:27:21.121702 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:21.121640    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-7bn5m\" (UniqueName: \"kubernetes.io/projected/c6373342-dd07-4497-a4f3-978ea9613b6e-kube-api-access-7bn5m\") pod \"console-operator-76dff48d59-5bfgg\" (UID: \"c6373342-dd07-4497-a4f3-978ea9613b6e\") " pod="openshift-console-operator/console-operator-76dff48d59-5bfgg"
May 26 11:27:21.121702 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:21.121682    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-xpkqm\" (UniqueName: \"kubernetes.io/projected/6aaefb25-f45b-44c1-8576-f0e9f4dd5977-kube-api-access-xpkqm\") pod \"volume-data-source-validator-5cf8c89d64-vsvjj\" (UID: \"6aaefb25-f45b-44c1-8576-f0e9f4dd5977\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5cf8c89d64-vsvjj"
May 26 11:27:21.121799 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:21.121709    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/97673f35-daf2-4eaf-b6cc-a119472f65c3-telemetry-config\") pod \"cluster-monitoring-operator-6c75959455-xqrk8\" (UID: \"97673f35-daf2-4eaf-b6cc-a119472f65c3\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-xqrk8"
May 26 11:27:21.122093 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:21.122076    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c6373342-dd07-4497-a4f3-978ea9613b6e-config\") pod \"console-operator-76dff48d59-5bfgg\" (UID: \"c6373342-dd07-4497-a4f3-978ea9613b6e\") " pod="openshift-console-operator/console-operator-76dff48d59-5bfgg"
May 26 11:27:21.122364 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:21.122345    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c6373342-dd07-4497-a4f3-978ea9613b6e-trusted-ca\") pod \"console-operator-76dff48d59-5bfgg\" (UID: \"c6373342-dd07-4497-a4f3-978ea9613b6e\") " pod="openshift-console-operator/console-operator-76dff48d59-5bfgg"
May 26 11:27:21.124642 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:21.124625    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c6373342-dd07-4497-a4f3-978ea9613b6e-serving-cert\") pod \"console-operator-76dff48d59-5bfgg\" (UID: \"c6373342-dd07-4497-a4f3-978ea9613b6e\") " pod="openshift-console-operator/console-operator-76dff48d59-5bfgg"
May 26 11:27:21.132013 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:21.131993    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7f44b6dd87-dpzf5"]
May 26 11:27:21.132692 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:21.132674    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-ds944"]
May 26 11:27:21.139710 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:21.139691    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpkqm\" (UniqueName: \"kubernetes.io/projected/6aaefb25-f45b-44c1-8576-f0e9f4dd5977-kube-api-access-xpkqm\") pod \"volume-data-source-validator-5cf8c89d64-vsvjj\" (UID: \"6aaefb25-f45b-44c1-8576-f0e9f4dd5977\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5cf8c89d64-vsvjj"
May 26 11:27:21.142765 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:21.142745    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bn5m\" (UniqueName: \"kubernetes.io/projected/c6373342-dd07-4497-a4f3-978ea9613b6e-kube-api-access-7bn5m\") pod \"console-operator-76dff48d59-5bfgg\" (UID: \"c6373342-dd07-4497-a4f3-978ea9613b6e\") " pod="openshift-console-operator/console-operator-76dff48d59-5bfgg"
May 26 11:27:21.164299 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:21.164258    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-57b5b68b8-sfz45"]
May 26 11:27:21.175688 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:21.175667    2572 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-78bdfb888c-g8l8m"]
May 26 11:27:21.179716 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:21.179699    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-78bdfb888c-g8l8m"
May 26 11:27:21.183522 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:21.183506    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
May 26 11:27:21.183739 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:21.183702    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-ppm6m\""
May 26 11:27:21.183896 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:21.183883    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
May 26 11:27:21.183980 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:21.183916    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
May 26 11:27:21.184177 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:21.184155    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
May 26 11:27:21.189139 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:21.189121    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
May 26 11:27:21.190854 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:21.190836    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-78bdfb888c-g8l8m"]
May 26 11:27:21.193580 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:21.193561    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5cf8c89d64-vsvjj"
May 26 11:27:21.222718 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:21.222695    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d2abca20-90fb-4503-b682-526829ccf2e0-config\") pod \"kube-storage-version-migrator-operator-565644f49-ds944\" (UID: \"d2abca20-90fb-4503-b682-526829ccf2e0\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-ds944"
May 26 11:27:21.222834 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:21.222730    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/33cb4d10-beab-4ba3-bde6-5ea8aad0a3bc-serving-cert\") pod \"insights-operator-78bdfb888c-g8l8m\" (UID: \"33cb4d10-beab-4ba3-bde6-5ea8aad0a3bc\") " pod="openshift-insights/insights-operator-78bdfb888c-g8l8m"
May 26 11:27:21.222834 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:21.222764    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/29497cac-89db-463d-81fe-5baccccba06c-serving-cert\") pod \"service-ca-operator-57b5b68b8-sfz45\" (UID: \"29497cac-89db-463d-81fe-5baccccba06c\") " pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-sfz45"
May 26 11:27:21.222834 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:21.222818    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/33cb4d10-beab-4ba3-bde6-5ea8aad0a3bc-snapshots\") pod \"insights-operator-78bdfb888c-g8l8m\" (UID: \"33cb4d10-beab-4ba3-bde6-5ea8aad0a3bc\") " pod="openshift-insights/insights-operator-78bdfb888c-g8l8m"
May 26 11:27:21.223011 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:21.222836    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wrrw\" (UniqueName: \"kubernetes.io/projected/3c1f1e68-2d05-4e41-a228-f123b7e838b0-kube-api-access-4wrrw\") pod \"network-check-source-7f44b6dd87-dpzf5\" (UID: \"3c1f1e68-2d05-4e41-a228-f123b7e838b0\") " pod="openshift-network-diagnostics/network-check-source-7f44b6dd87-dpzf5"
May 26 11:27:21.223011 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:21.222861    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d2abca20-90fb-4503-b682-526829ccf2e0-serving-cert\") pod \"kube-storage-version-migrator-operator-565644f49-ds944\" (UID: \"d2abca20-90fb-4503-b682-526829ccf2e0\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-ds944"
May 26 11:27:21.223011 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:21.222877    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjb5j\" (UniqueName: \"kubernetes.io/projected/d2abca20-90fb-4503-b682-526829ccf2e0-kube-api-access-kjb5j\") pod \"kube-storage-version-migrator-operator-565644f49-ds944\" (UID: \"d2abca20-90fb-4503-b682-526829ccf2e0\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-ds944"
May 26 11:27:21.223011 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:21.222909    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/33cb4d10-beab-4ba3-bde6-5ea8aad0a3bc-tmp\") pod \"insights-operator-78bdfb888c-g8l8m\" (UID: \"33cb4d10-beab-4ba3-bde6-5ea8aad0a3bc\") " pod="openshift-insights/insights-operator-78bdfb888c-g8l8m"
May 26 11:27:21.223011 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:21.222943    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/33cb4d10-beab-4ba3-bde6-5ea8aad0a3bc-trusted-ca-bundle\") pod \"insights-operator-78bdfb888c-g8l8m\" (UID: \"33cb4d10-beab-4ba3-bde6-5ea8aad0a3bc\") " pod="openshift-insights/insights-operator-78bdfb888c-g8l8m"
May 26 11:27:21.223011 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:21.222962    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/33cb4d10-beab-4ba3-bde6-5ea8aad0a3bc-service-ca-bundle\") pod \"insights-operator-78bdfb888c-g8l8m\" (UID: \"33cb4d10-beab-4ba3-bde6-5ea8aad0a3bc\") " pod="openshift-insights/insights-operator-78bdfb888c-g8l8m"
May 26 11:27:21.223011 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:21.222978    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtqlx\" (UniqueName: \"kubernetes.io/projected/33cb4d10-beab-4ba3-bde6-5ea8aad0a3bc-kube-api-access-qtqlx\") pod \"insights-operator-78bdfb888c-g8l8m\" (UID: \"33cb4d10-beab-4ba3-bde6-5ea8aad0a3bc\") " pod="openshift-insights/insights-operator-78bdfb888c-g8l8m"
May 26 11:27:21.223011 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:21.222999    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghbcl\" (UniqueName: \"kubernetes.io/projected/29497cac-89db-463d-81fe-5baccccba06c-kube-api-access-ghbcl\") pod \"service-ca-operator-57b5b68b8-sfz45\" (UID: \"29497cac-89db-463d-81fe-5baccccba06c\") " pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-sfz45"
May 26 11:27:21.223269 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:21.223053    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/29497cac-89db-463d-81fe-5baccccba06c-config\") pod \"service-ca-operator-57b5b68b8-sfz45\" (UID: \"29497cac-89db-463d-81fe-5baccccba06c\") " pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-sfz45"
May 26 11:27:21.293409 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:21.293383    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-76dff48d59-5bfgg"
May 26 11:27:21.321177 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:21.321149    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5cf8c89d64-vsvjj"]
May 26 11:27:21.323276 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:27:21.323247    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6aaefb25_f45b_44c1_8576_f0e9f4dd5977.slice/crio-41ed572aeed60138eb34622b650e18580a2d8c602891693326a3380b8d8a7e67 WatchSource:0}: Error finding container 41ed572aeed60138eb34622b650e18580a2d8c602891693326a3380b8d8a7e67: Status 404 returned error can't find the container with id 41ed572aeed60138eb34622b650e18580a2d8c602891693326a3380b8d8a7e67
May 26 11:27:21.323373 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:21.323336    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/33cb4d10-beab-4ba3-bde6-5ea8aad0a3bc-tmp\") pod \"insights-operator-78bdfb888c-g8l8m\" (UID: \"33cb4d10-beab-4ba3-bde6-5ea8aad0a3bc\") " pod="openshift-insights/insights-operator-78bdfb888c-g8l8m"
May 26 11:27:21.323373 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:21.323364    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/33cb4d10-beab-4ba3-bde6-5ea8aad0a3bc-trusted-ca-bundle\") pod \"insights-operator-78bdfb888c-g8l8m\" (UID: \"33cb4d10-beab-4ba3-bde6-5ea8aad0a3bc\") " pod="openshift-insights/insights-operator-78bdfb888c-g8l8m"
May 26 11:27:21.323460 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:21.323385    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/33cb4d10-beab-4ba3-bde6-5ea8aad0a3bc-service-ca-bundle\") pod \"insights-operator-78bdfb888c-g8l8m\" (UID: \"33cb4d10-beab-4ba3-bde6-5ea8aad0a3bc\") " pod="openshift-insights/insights-operator-78bdfb888c-g8l8m"
May 26 11:27:21.323460 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:21.323404    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-qtqlx\" (UniqueName: \"kubernetes.io/projected/33cb4d10-beab-4ba3-bde6-5ea8aad0a3bc-kube-api-access-qtqlx\") pod \"insights-operator-78bdfb888c-g8l8m\" (UID: \"33cb4d10-beab-4ba3-bde6-5ea8aad0a3bc\") " pod="openshift-insights/insights-operator-78bdfb888c-g8l8m"
May 26 11:27:21.323460 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:21.323430    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-ghbcl\" (UniqueName: \"kubernetes.io/projected/29497cac-89db-463d-81fe-5baccccba06c-kube-api-access-ghbcl\") pod \"service-ca-operator-57b5b68b8-sfz45\" (UID: \"29497cac-89db-463d-81fe-5baccccba06c\") " pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-sfz45"
May 26 11:27:21.323605 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:21.323581    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/29497cac-89db-463d-81fe-5baccccba06c-config\") pod \"service-ca-operator-57b5b68b8-sfz45\" (UID: \"29497cac-89db-463d-81fe-5baccccba06c\") " pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-sfz45"
May 26 11:27:21.323664 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:21.323630    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d2abca20-90fb-4503-b682-526829ccf2e0-config\") pod \"kube-storage-version-migrator-operator-565644f49-ds944\" (UID: \"d2abca20-90fb-4503-b682-526829ccf2e0\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-ds944"
May 26 11:27:21.323729 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:21.323672    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/33cb4d10-beab-4ba3-bde6-5ea8aad0a3bc-serving-cert\") pod \"insights-operator-78bdfb888c-g8l8m\" (UID: \"33cb4d10-beab-4ba3-bde6-5ea8aad0a3bc\") " pod="openshift-insights/insights-operator-78bdfb888c-g8l8m"
May 26 11:27:21.323780 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:21.323736    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/29497cac-89db-463d-81fe-5baccccba06c-serving-cert\") pod \"service-ca-operator-57b5b68b8-sfz45\" (UID: \"29497cac-89db-463d-81fe-5baccccba06c\") " pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-sfz45"
May 26 11:27:21.323836 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:21.323805    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/33cb4d10-beab-4ba3-bde6-5ea8aad0a3bc-snapshots\") pod \"insights-operator-78bdfb888c-g8l8m\" (UID: \"33cb4d10-beab-4ba3-bde6-5ea8aad0a3bc\") " pod="openshift-insights/insights-operator-78bdfb888c-g8l8m"
May 26 11:27:21.323891 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:21.323837    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-4wrrw\" (UniqueName: \"kubernetes.io/projected/3c1f1e68-2d05-4e41-a228-f123b7e838b0-kube-api-access-4wrrw\") pod \"network-check-source-7f44b6dd87-dpzf5\" (UID: \"3c1f1e68-2d05-4e41-a228-f123b7e838b0\") " pod="openshift-network-diagnostics/network-check-source-7f44b6dd87-dpzf5"
May 26 11:27:21.323891 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:21.323881    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d2abca20-90fb-4503-b682-526829ccf2e0-serving-cert\") pod \"kube-storage-version-migrator-operator-565644f49-ds944\" (UID: \"d2abca20-90fb-4503-b682-526829ccf2e0\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-ds944"
May 26 11:27:21.323980 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:21.323908    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-kjb5j\" (UniqueName: \"kubernetes.io/projected/d2abca20-90fb-4503-b682-526829ccf2e0-kube-api-access-kjb5j\") pod \"kube-storage-version-migrator-operator-565644f49-ds944\" (UID: \"d2abca20-90fb-4503-b682-526829ccf2e0\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-ds944"
May 26 11:27:21.324076 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:21.324006    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/33cb4d10-beab-4ba3-bde6-5ea8aad0a3bc-service-ca-bundle\") pod \"insights-operator-78bdfb888c-g8l8m\" (UID: \"33cb4d10-beab-4ba3-bde6-5ea8aad0a3bc\") " pod="openshift-insights/insights-operator-78bdfb888c-g8l8m"
May 26 11:27:21.326924 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:21.324886    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/33cb4d10-beab-4ba3-bde6-5ea8aad0a3bc-snapshots\") pod \"insights-operator-78bdfb888c-g8l8m\" (UID: \"33cb4d10-beab-4ba3-bde6-5ea8aad0a3bc\") " pod="openshift-insights/insights-operator-78bdfb888c-g8l8m"
May 26 11:27:21.326924 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:21.325490    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/29497cac-89db-463d-81fe-5baccccba06c-config\") pod \"service-ca-operator-57b5b68b8-sfz45\" (UID: \"29497cac-89db-463d-81fe-5baccccba06c\") " pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-sfz45"
May 26 11:27:21.326924 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:21.323734    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/33cb4d10-beab-4ba3-bde6-5ea8aad0a3bc-tmp\") pod \"insights-operator-78bdfb888c-g8l8m\" (UID: \"33cb4d10-beab-4ba3-bde6-5ea8aad0a3bc\") " pod="openshift-insights/insights-operator-78bdfb888c-g8l8m"
May 26 11:27:21.326924 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:21.325530    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d2abca20-90fb-4503-b682-526829ccf2e0-config\") pod \"kube-storage-version-migrator-operator-565644f49-ds944\" (UID: \"d2abca20-90fb-4503-b682-526829ccf2e0\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-ds944"
May 26 11:27:21.326924 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:21.325758    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/33cb4d10-beab-4ba3-bde6-5ea8aad0a3bc-trusted-ca-bundle\") pod \"insights-operator-78bdfb888c-g8l8m\" (UID: \"33cb4d10-beab-4ba3-bde6-5ea8aad0a3bc\") " pod="openshift-insights/insights-operator-78bdfb888c-g8l8m"
May 26 11:27:21.326924 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:21.326857    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/33cb4d10-beab-4ba3-bde6-5ea8aad0a3bc-serving-cert\") pod \"insights-operator-78bdfb888c-g8l8m\" (UID: \"33cb4d10-beab-4ba3-bde6-5ea8aad0a3bc\") " pod="openshift-insights/insights-operator-78bdfb888c-g8l8m"
May 26 11:27:21.329357 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:21.327481    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d2abca20-90fb-4503-b682-526829ccf2e0-serving-cert\") pod \"kube-storage-version-migrator-operator-565644f49-ds944\" (UID: \"d2abca20-90fb-4503-b682-526829ccf2e0\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-ds944"
May 26 11:27:21.331597 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:21.331498    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/29497cac-89db-463d-81fe-5baccccba06c-serving-cert\") pod \"service-ca-operator-57b5b68b8-sfz45\" (UID: \"29497cac-89db-463d-81fe-5baccccba06c\") " pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-sfz45"
May 26 11:27:21.336417 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:21.336392    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghbcl\" (UniqueName: \"kubernetes.io/projected/29497cac-89db-463d-81fe-5baccccba06c-kube-api-access-ghbcl\") pod \"service-ca-operator-57b5b68b8-sfz45\" (UID: \"29497cac-89db-463d-81fe-5baccccba06c\") " pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-sfz45"
May 26 11:27:21.338049 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:21.338027    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wrrw\" (UniqueName: \"kubernetes.io/projected/3c1f1e68-2d05-4e41-a228-f123b7e838b0-kube-api-access-4wrrw\") pod \"network-check-source-7f44b6dd87-dpzf5\" (UID: \"3c1f1e68-2d05-4e41-a228-f123b7e838b0\") " pod="openshift-network-diagnostics/network-check-source-7f44b6dd87-dpzf5"
May 26 11:27:21.338396 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:21.338378    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtqlx\" (UniqueName: \"kubernetes.io/projected/33cb4d10-beab-4ba3-bde6-5ea8aad0a3bc-kube-api-access-qtqlx\") pod \"insights-operator-78bdfb888c-g8l8m\" (UID: \"33cb4d10-beab-4ba3-bde6-5ea8aad0a3bc\") " pod="openshift-insights/insights-operator-78bdfb888c-g8l8m"
May 26 11:27:21.342445 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:21.342418    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjb5j\" (UniqueName: \"kubernetes.io/projected/d2abca20-90fb-4503-b682-526829ccf2e0-kube-api-access-kjb5j\") pod \"kube-storage-version-migrator-operator-565644f49-ds944\" (UID: \"d2abca20-90fb-4503-b682-526829ccf2e0\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-ds944"
May 26 11:27:21.400608 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:21.400581    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-sfz45"
May 26 11:27:21.407357 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:21.407334    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7f44b6dd87-dpzf5"
May 26 11:27:21.412012 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:21.411983    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-ds944"
May 26 11:27:21.430238 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:21.430214    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-76dff48d59-5bfgg"]
May 26 11:27:21.432673 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:27:21.432579    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6373342_dd07_4497_a4f3_978ea9613b6e.slice/crio-bf742c2fd523ac2083c75dcf98406bb9bb9a26e352674236f96173ecbc85ef5c WatchSource:0}: Error finding container bf742c2fd523ac2083c75dcf98406bb9bb9a26e352674236f96173ecbc85ef5c: Status 404 returned error can't find the container with id bf742c2fd523ac2083c75dcf98406bb9bb9a26e352674236f96173ecbc85ef5c
May 26 11:27:21.488411 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:21.488364    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-78bdfb888c-g8l8m"
May 26 11:27:21.526865 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:21.526830    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ed80247a-4151-4422-b05e-a1764ec5d650-samples-operator-tls\") pod \"cluster-samples-operator-697587599-hls8q\" (UID: \"ed80247a-4151-4422-b05e-a1764ec5d650\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-hls8q"
May 26 11:27:21.527038 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:27:21.527005    2572 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
May 26 11:27:21.527141 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:27:21.527092    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed80247a-4151-4422-b05e-a1764ec5d650-samples-operator-tls podName:ed80247a-4151-4422-b05e-a1764ec5d650 nodeName:}" failed. No retries permitted until 2026-05-26 11:27:22.527068057 +0000 UTC m=+165.429278374 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/ed80247a-4151-4422-b05e-a1764ec5d650-samples-operator-tls") pod "cluster-samples-operator-697587599-hls8q" (UID: "ed80247a-4151-4422-b05e-a1764ec5d650") : secret "samples-operator-tls" not found
May 26 11:27:21.549211 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:21.549132    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-57b5b68b8-sfz45"]
May 26 11:27:21.555284 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:27:21.555249    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29497cac_89db_463d_81fe_5baccccba06c.slice/crio-9873b304d417a5e9e97faa8461cdf346f317ab9ac26aa0503c8b97d5f096dd5f WatchSource:0}: Error finding container 9873b304d417a5e9e97faa8461cdf346f317ab9ac26aa0503c8b97d5f096dd5f: Status 404 returned error can't find the container with id 9873b304d417a5e9e97faa8461cdf346f317ab9ac26aa0503c8b97d5f096dd5f
May 26 11:27:21.635105 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:21.635039    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-78bdfb888c-g8l8m"]
May 26 11:27:21.637614 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:27:21.637589    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33cb4d10_beab_4ba3_bde6_5ea8aad0a3bc.slice/crio-e04cab98dc38b3c37cf8eaa402c0553fe67cddc296457f186afec3c7383f5bf9 WatchSource:0}: Error finding container e04cab98dc38b3c37cf8eaa402c0553fe67cddc296457f186afec3c7383f5bf9: Status 404 returned error can't find the container with id e04cab98dc38b3c37cf8eaa402c0553fe67cddc296457f186afec3c7383f5bf9
May 26 11:27:21.771150 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:21.771130    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7f44b6dd87-dpzf5"]
May 26 11:27:21.773895 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:21.773860    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-ds944"]
May 26 11:27:21.775055 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:27:21.774554    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c1f1e68_2d05_4e41_a228_f123b7e838b0.slice/crio-646773892ef389fd67ca32756086c50f643295bd5a2a6e4b3ead3499ca445430 WatchSource:0}: Error finding container 646773892ef389fd67ca32756086c50f643295bd5a2a6e4b3ead3499ca445430: Status 404 returned error can't find the container with id 646773892ef389fd67ca32756086c50f643295bd5a2a6e4b3ead3499ca445430
May 26 11:27:21.778876 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:27:21.778853    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2abca20_90fb_4503_b682_526829ccf2e0.slice/crio-5549236a8f5c343abe9b9f364dd2fd7010dec0c7adc76df8f06a237d76a34ec1 WatchSource:0}: Error finding container 5549236a8f5c343abe9b9f364dd2fd7010dec0c7adc76df8f06a237d76a34ec1: Status 404 returned error can't find the container with id 5549236a8f5c343abe9b9f364dd2fd7010dec0c7adc76df8f06a237d76a34ec1
May 26 11:27:22.122514 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:27:22.122481    2572 configmap.go:193] Couldn't get configMap openshift-monitoring/telemetry-config: failed to sync configmap cache: timed out waiting for the condition
May 26 11:27:22.123010 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:27:22.122580    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/97673f35-daf2-4eaf-b6cc-a119472f65c3-telemetry-config podName:97673f35-daf2-4eaf-b6cc-a119472f65c3 nodeName:}" failed. No retries permitted until 2026-05-26 11:27:22.622558856 +0000 UTC m=+165.524769176 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "telemetry-config" (UniqueName: "kubernetes.io/configmap/97673f35-daf2-4eaf-b6cc-a119472f65c3-telemetry-config") pod "cluster-monitoring-operator-6c75959455-xqrk8" (UID: "97673f35-daf2-4eaf-b6cc-a119472f65c3") : failed to sync configmap cache: timed out waiting for the condition
May 26 11:27:22.123010 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:27:22.122839    2572 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: failed to sync secret cache: timed out waiting for the condition
May 26 11:27:22.123010 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:27:22.122888    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/97673f35-daf2-4eaf-b6cc-a119472f65c3-cluster-monitoring-operator-tls podName:97673f35-daf2-4eaf-b6cc-a119472f65c3 nodeName:}" failed. No retries permitted until 2026-05-26 11:27:22.622871739 +0000 UTC m=+165.525082058 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/97673f35-daf2-4eaf-b6cc-a119472f65c3-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6c75959455-xqrk8" (UID: "97673f35-daf2-4eaf-b6cc-a119472f65c3") : failed to sync secret cache: timed out waiting for the condition
May 26 11:27:22.137782 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:27:22.137757    2572 projected.go:289] Couldn't get configMap openshift-monitoring/openshift-service-ca.crt: failed to sync configmap cache: timed out waiting for the condition
May 26 11:27:22.137782 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:27:22.137781    2572 projected.go:194] Error preparing data for projected volume kube-api-access-gd6zq for pod openshift-monitoring/cluster-monitoring-operator-6c75959455-xqrk8: failed to sync configmap cache: timed out waiting for the condition
May 26 11:27:22.137978 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:27:22.137841    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/97673f35-daf2-4eaf-b6cc-a119472f65c3-kube-api-access-gd6zq podName:97673f35-daf2-4eaf-b6cc-a119472f65c3 nodeName:}" failed. No retries permitted until 2026-05-26 11:27:22.637824345 +0000 UTC m=+165.540034669 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-gd6zq" (UniqueName: "kubernetes.io/projected/97673f35-daf2-4eaf-b6cc-a119472f65c3-kube-api-access-gd6zq") pod "cluster-monitoring-operator-6c75959455-xqrk8" (UID: "97673f35-daf2-4eaf-b6cc-a119472f65c3") : failed to sync configmap cache: timed out waiting for the condition
May 26 11:27:22.200949 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:22.200883    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-78bdfb888c-g8l8m" event={"ID":"33cb4d10-beab-4ba3-bde6-5ea8aad0a3bc","Type":"ContainerStarted","Data":"e04cab98dc38b3c37cf8eaa402c0553fe67cddc296457f186afec3c7383f5bf9"}
May 26 11:27:22.203042 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:22.202978    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-76dff48d59-5bfgg" event={"ID":"c6373342-dd07-4497-a4f3-978ea9613b6e","Type":"ContainerStarted","Data":"bf742c2fd523ac2083c75dcf98406bb9bb9a26e352674236f96173ecbc85ef5c"}
May 26 11:27:22.205860 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:22.205828    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5cf8c89d64-vsvjj" event={"ID":"6aaefb25-f45b-44c1-8576-f0e9f4dd5977","Type":"ContainerStarted","Data":"41ed572aeed60138eb34622b650e18580a2d8c602891693326a3380b8d8a7e67"}
May 26 11:27:22.207935 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:22.207912    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-ds944" event={"ID":"d2abca20-90fb-4503-b682-526829ccf2e0","Type":"ContainerStarted","Data":"5549236a8f5c343abe9b9f364dd2fd7010dec0c7adc76df8f06a237d76a34ec1"}
May 26 11:27:22.209856 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:22.209799    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7f44b6dd87-dpzf5" event={"ID":"3c1f1e68-2d05-4e41-a228-f123b7e838b0","Type":"ContainerStarted","Data":"e4783bb47dfb0990cb308425b0a27ffe67a6b2251a66e9867488e621d9c5fbe1"}
May 26 11:27:22.209856 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:22.209830    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7f44b6dd87-dpzf5" event={"ID":"3c1f1e68-2d05-4e41-a228-f123b7e838b0","Type":"ContainerStarted","Data":"646773892ef389fd67ca32756086c50f643295bd5a2a6e4b3ead3499ca445430"}
May 26 11:27:22.211051 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:22.211018    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-sfz45" event={"ID":"29497cac-89db-463d-81fe-5baccccba06c","Type":"ContainerStarted","Data":"9873b304d417a5e9e97faa8461cdf346f317ab9ac26aa0503c8b97d5f096dd5f"}
May 26 11:27:22.239291 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:22.239077    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
May 26 11:27:22.239396 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:22.239330    2572 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-7f44b6dd87-dpzf5" podStartSLOduration=1.2393170119999999 podStartE2EDuration="1.239317012s" podCreationTimestamp="2026-05-26 11:27:21 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 11:27:22.238127562 +0000 UTC m=+165.140337892" watchObservedRunningTime="2026-05-26 11:27:22.239317012 +0000 UTC m=+165.141527345"
May 26 11:27:22.299601 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:22.299571    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
May 26 11:27:22.535972 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:22.535938    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ed80247a-4151-4422-b05e-a1764ec5d650-samples-operator-tls\") pod \"cluster-samples-operator-697587599-hls8q\" (UID: \"ed80247a-4151-4422-b05e-a1764ec5d650\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-hls8q"
May 26 11:27:22.536163 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:27:22.536097    2572 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
May 26 11:27:22.536163 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:27:22.536161    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed80247a-4151-4422-b05e-a1764ec5d650-samples-operator-tls podName:ed80247a-4151-4422-b05e-a1764ec5d650 nodeName:}" failed. No retries permitted until 2026-05-26 11:27:24.536142498 +0000 UTC m=+167.438352818 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/ed80247a-4151-4422-b05e-a1764ec5d650-samples-operator-tls") pod "cluster-samples-operator-697587599-hls8q" (UID: "ed80247a-4151-4422-b05e-a1764ec5d650") : secret "samples-operator-tls" not found
May 26 11:27:22.573028 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:22.572804    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
May 26 11:27:22.636814 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:22.636781    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/97673f35-daf2-4eaf-b6cc-a119472f65c3-telemetry-config\") pod \"cluster-monitoring-operator-6c75959455-xqrk8\" (UID: \"97673f35-daf2-4eaf-b6cc-a119472f65c3\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-xqrk8"
May 26 11:27:22.636980 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:22.636915    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/97673f35-daf2-4eaf-b6cc-a119472f65c3-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6c75959455-xqrk8\" (UID: \"97673f35-daf2-4eaf-b6cc-a119472f65c3\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-xqrk8"
May 26 11:27:22.637075 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:27:22.637055    2572 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
May 26 11:27:22.637136 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:27:22.637124    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/97673f35-daf2-4eaf-b6cc-a119472f65c3-cluster-monitoring-operator-tls podName:97673f35-daf2-4eaf-b6cc-a119472f65c3 nodeName:}" failed. No retries permitted until 2026-05-26 11:27:23.63710378 +0000 UTC m=+166.539314101 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/97673f35-daf2-4eaf-b6cc-a119472f65c3-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6c75959455-xqrk8" (UID: "97673f35-daf2-4eaf-b6cc-a119472f65c3") : secret "cluster-monitoring-operator-tls" not found
May 26 11:27:22.637721 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:22.637693    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/97673f35-daf2-4eaf-b6cc-a119472f65c3-telemetry-config\") pod \"cluster-monitoring-operator-6c75959455-xqrk8\" (UID: \"97673f35-daf2-4eaf-b6cc-a119472f65c3\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-xqrk8"
May 26 11:27:22.737458 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:22.737423    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-gd6zq\" (UniqueName: \"kubernetes.io/projected/97673f35-daf2-4eaf-b6cc-a119472f65c3-kube-api-access-gd6zq\") pod \"cluster-monitoring-operator-6c75959455-xqrk8\" (UID: \"97673f35-daf2-4eaf-b6cc-a119472f65c3\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-xqrk8"
May 26 11:27:22.742482 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:22.742431    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gd6zq\" (UniqueName: \"kubernetes.io/projected/97673f35-daf2-4eaf-b6cc-a119472f65c3-kube-api-access-gd6zq\") pod \"cluster-monitoring-operator-6c75959455-xqrk8\" (UID: \"97673f35-daf2-4eaf-b6cc-a119472f65c3\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-xqrk8"
May 26 11:27:23.646566 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:23.646520    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/97673f35-daf2-4eaf-b6cc-a119472f65c3-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6c75959455-xqrk8\" (UID: \"97673f35-daf2-4eaf-b6cc-a119472f65c3\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-xqrk8"
May 26 11:27:23.646909 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:27:23.646687    2572 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
May 26 11:27:23.646909 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:27:23.646770    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/97673f35-daf2-4eaf-b6cc-a119472f65c3-cluster-monitoring-operator-tls podName:97673f35-daf2-4eaf-b6cc-a119472f65c3 nodeName:}" failed. No retries permitted until 2026-05-26 11:27:25.646749642 +0000 UTC m=+168.548959964 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/97673f35-daf2-4eaf-b6cc-a119472f65c3-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6c75959455-xqrk8" (UID: "97673f35-daf2-4eaf-b6cc-a119472f65c3") : secret "cluster-monitoring-operator-tls" not found
May 26 11:27:24.553109 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:24.553071    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ed80247a-4151-4422-b05e-a1764ec5d650-samples-operator-tls\") pod \"cluster-samples-operator-697587599-hls8q\" (UID: \"ed80247a-4151-4422-b05e-a1764ec5d650\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-hls8q"
May 26 11:27:24.553297 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:27:24.553256    2572 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
May 26 11:27:24.553367 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:27:24.553336    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed80247a-4151-4422-b05e-a1764ec5d650-samples-operator-tls podName:ed80247a-4151-4422-b05e-a1764ec5d650 nodeName:}" failed. No retries permitted until 2026-05-26 11:27:28.553314513 +0000 UTC m=+171.455524828 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/ed80247a-4151-4422-b05e-a1764ec5d650-samples-operator-tls") pod "cluster-samples-operator-697587599-hls8q" (UID: "ed80247a-4151-4422-b05e-a1764ec5d650") : secret "samples-operator-tls" not found
May 26 11:27:25.662175 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:25.662137    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/97673f35-daf2-4eaf-b6cc-a119472f65c3-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6c75959455-xqrk8\" (UID: \"97673f35-daf2-4eaf-b6cc-a119472f65c3\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-xqrk8"
May 26 11:27:25.662615 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:27:25.662308    2572 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
May 26 11:27:25.662615 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:27:25.662390    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/97673f35-daf2-4eaf-b6cc-a119472f65c3-cluster-monitoring-operator-tls podName:97673f35-daf2-4eaf-b6cc-a119472f65c3 nodeName:}" failed. No retries permitted until 2026-05-26 11:27:29.662370257 +0000 UTC m=+172.564580570 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/97673f35-daf2-4eaf-b6cc-a119472f65c3-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6c75959455-xqrk8" (UID: "97673f35-daf2-4eaf-b6cc-a119472f65c3") : secret "cluster-monitoring-operator-tls" not found
May 26 11:27:26.227608 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:26.227572    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-ds944" event={"ID":"d2abca20-90fb-4503-b682-526829ccf2e0","Type":"ContainerStarted","Data":"787e112f98fa0f8689107e53260eaef9a446fee18fb3f59fdb04f0775e1727c4"}
May 26 11:27:26.228917 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:26.228880    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-sfz45" event={"ID":"29497cac-89db-463d-81fe-5baccccba06c","Type":"ContainerStarted","Data":"d0f6b755dc2935c76741779e85f377650d229dff553f09124287266ebecd4221"}
May 26 11:27:26.230176 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:26.230149    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-78bdfb888c-g8l8m" event={"ID":"33cb4d10-beab-4ba3-bde6-5ea8aad0a3bc","Type":"ContainerStarted","Data":"14ff270d8eef02afad6d7489bea9d419ddf2308d0354b1320f6bc39c22059b64"}
May 26 11:27:26.231513 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:26.231495    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-5bfgg_c6373342-dd07-4497-a4f3-978ea9613b6e/console-operator/0.log"
May 26 11:27:26.231625 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:26.231525    2572 generic.go:358] "Generic (PLEG): container finished" podID="c6373342-dd07-4497-a4f3-978ea9613b6e" containerID="49562ffc5da9534067302750d41437ca8eb88c0ab3ef944940fe6192c45d429c" exitCode=255
May 26 11:27:26.231625 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:26.231550    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-76dff48d59-5bfgg" event={"ID":"c6373342-dd07-4497-a4f3-978ea9613b6e","Type":"ContainerDied","Data":"49562ffc5da9534067302750d41437ca8eb88c0ab3ef944940fe6192c45d429c"}
May 26 11:27:26.231817 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:26.231799    2572 scope.go:117] "RemoveContainer" containerID="49562ffc5da9534067302750d41437ca8eb88c0ab3ef944940fe6192c45d429c"
May 26 11:27:26.232902 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:26.232882    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5cf8c89d64-vsvjj" event={"ID":"6aaefb25-f45b-44c1-8576-f0e9f4dd5977","Type":"ContainerStarted","Data":"820eee23c4b0b8223b52fee8d847db0290eddd0fcf4f61c892f0491014a35b4e"}
May 26 11:27:26.273439 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:26.273387    2572 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-ds944" podStartSLOduration=1.8460151200000001 podStartE2EDuration="5.273372263s" podCreationTimestamp="2026-05-26 11:27:21 +0000 UTC" firstStartedPulling="2026-05-26 11:27:21.781622939 +0000 UTC m=+164.683833251" lastFinishedPulling="2026-05-26 11:27:25.208980087 +0000 UTC m=+168.111190394" observedRunningTime="2026-05-26 11:27:26.271773792 +0000 UTC m=+169.173984123" watchObservedRunningTime="2026-05-26 11:27:26.273372263 +0000 UTC m=+169.175582595"
May 26 11:27:26.325312 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:26.325257    2572 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-5cf8c89d64-vsvjj" podStartSLOduration=4.655911891 podStartE2EDuration="6.325236995s" podCreationTimestamp="2026-05-26 11:27:20 +0000 UTC" firstStartedPulling="2026-05-26 11:27:21.331134566 +0000 UTC m=+164.233344873" lastFinishedPulling="2026-05-26 11:27:23.000459648 +0000 UTC m=+165.902669977" observedRunningTime="2026-05-26 11:27:26.324252037 +0000 UTC m=+169.226462367" watchObservedRunningTime="2026-05-26 11:27:26.325236995 +0000 UTC m=+169.227447324"
May 26 11:27:26.383061 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:26.383009    2572 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-insights/insights-operator-78bdfb888c-g8l8m" podStartSLOduration=1.8209824970000001 podStartE2EDuration="5.382993638s" podCreationTimestamp="2026-05-26 11:27:21 +0000 UTC" firstStartedPulling="2026-05-26 11:27:21.639359439 +0000 UTC m=+164.541569749" lastFinishedPulling="2026-05-26 11:27:25.201370572 +0000 UTC m=+168.103580890" observedRunningTime="2026-05-26 11:27:26.381769171 +0000 UTC m=+169.283979500" watchObservedRunningTime="2026-05-26 11:27:26.382993638 +0000 UTC m=+169.285203967"
May 26 11:27:26.416919 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:26.416872    2572 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-sfz45" podStartSLOduration=1.772926724 podStartE2EDuration="5.416857446s" podCreationTimestamp="2026-05-26 11:27:21 +0000 UTC" firstStartedPulling="2026-05-26 11:27:21.557110334 +0000 UTC m=+164.459320645" lastFinishedPulling="2026-05-26 11:27:25.201041051 +0000 UTC m=+168.103251367" observedRunningTime="2026-05-26 11:27:26.416301872 +0000 UTC m=+169.318512203" watchObservedRunningTime="2026-05-26 11:27:26.416857446 +0000 UTC m=+169.319067774"
May 26 11:27:27.236877 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:27.236849    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-5bfgg_c6373342-dd07-4497-a4f3-978ea9613b6e/console-operator/1.log"
May 26 11:27:27.237279 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:27.237213    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-5bfgg_c6373342-dd07-4497-a4f3-978ea9613b6e/console-operator/0.log"
May 26 11:27:27.237279 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:27.237246    2572 generic.go:358] "Generic (PLEG): container finished" podID="c6373342-dd07-4497-a4f3-978ea9613b6e" containerID="361a17845f710634401f0d559637fd64ef786ffebc338043bdcdbdc6003ca836" exitCode=255
May 26 11:27:27.237377 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:27.237274    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-76dff48d59-5bfgg" event={"ID":"c6373342-dd07-4497-a4f3-978ea9613b6e","Type":"ContainerDied","Data":"361a17845f710634401f0d559637fd64ef786ffebc338043bdcdbdc6003ca836"}
May 26 11:27:27.237377 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:27.237324    2572 scope.go:117] "RemoveContainer" containerID="49562ffc5da9534067302750d41437ca8eb88c0ab3ef944940fe6192c45d429c"
May 26 11:27:27.238027 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:27.238006    2572 scope.go:117] "RemoveContainer" containerID="361a17845f710634401f0d559637fd64ef786ffebc338043bdcdbdc6003ca836"
May 26 11:27:27.238176 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:27:27.238154    2572 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-76dff48d59-5bfgg_openshift-console-operator(c6373342-dd07-4497-a4f3-978ea9613b6e)\"" pod="openshift-console-operator/console-operator-76dff48d59-5bfgg" podUID="c6373342-dd07-4497-a4f3-978ea9613b6e"
May 26 11:27:27.672225 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:27.672182    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5898c6cd7c-xxs9f"
May 26 11:27:27.980557 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:27.980487    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-4clzr_024ce274-4936-49fa-b084-ec86c46d34ed/dns-node-resolver/0.log"
May 26 11:27:28.241203 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:28.241122    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-5bfgg_c6373342-dd07-4497-a4f3-978ea9613b6e/console-operator/1.log"
May 26 11:27:28.241553 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:28.241474    2572 scope.go:117] "RemoveContainer" containerID="361a17845f710634401f0d559637fd64ef786ffebc338043bdcdbdc6003ca836"
May 26 11:27:28.241675 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:27:28.241656    2572 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-76dff48d59-5bfgg_openshift-console-operator(c6373342-dd07-4497-a4f3-978ea9613b6e)\"" pod="openshift-console-operator/console-operator-76dff48d59-5bfgg" podUID="c6373342-dd07-4497-a4f3-978ea9613b6e"
May 26 11:27:28.586650 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:28.586567    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ed80247a-4151-4422-b05e-a1764ec5d650-samples-operator-tls\") pod \"cluster-samples-operator-697587599-hls8q\" (UID: \"ed80247a-4151-4422-b05e-a1764ec5d650\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-hls8q"
May 26 11:27:28.586781 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:27:28.586709    2572 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
May 26 11:27:28.586781 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:27:28.586770    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed80247a-4151-4422-b05e-a1764ec5d650-samples-operator-tls podName:ed80247a-4151-4422-b05e-a1764ec5d650 nodeName:}" failed. No retries permitted until 2026-05-26 11:27:36.586753937 +0000 UTC m=+179.488964252 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/ed80247a-4151-4422-b05e-a1764ec5d650-samples-operator-tls") pod "cluster-samples-operator-697587599-hls8q" (UID: "ed80247a-4151-4422-b05e-a1764ec5d650") : secret "samples-operator-tls" not found
May 26 11:27:28.671472 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:28.671443    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-m6p64"
May 26 11:27:29.379859 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:29.379832    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-s9s2b_240a9693-e4b9-485c-a138-6c3eb7f6734a/node-ca/0.log"
May 26 11:27:29.693785 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:29.693747    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/97673f35-daf2-4eaf-b6cc-a119472f65c3-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6c75959455-xqrk8\" (UID: \"97673f35-daf2-4eaf-b6cc-a119472f65c3\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-xqrk8"
May 26 11:27:29.693967 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:27:29.693888    2572 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
May 26 11:27:29.693967 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:27:29.693947    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/97673f35-daf2-4eaf-b6cc-a119472f65c3-cluster-monitoring-operator-tls podName:97673f35-daf2-4eaf-b6cc-a119472f65c3 nodeName:}" failed. No retries permitted until 2026-05-26 11:27:37.693929941 +0000 UTC m=+180.596140248 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/97673f35-daf2-4eaf-b6cc-a119472f65c3-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6c75959455-xqrk8" (UID: "97673f35-daf2-4eaf-b6cc-a119472f65c3") : secret "cluster-monitoring-operator-tls" not found
May 26 11:27:30.670542 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:30.670508    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k7jgt"
May 26 11:27:31.294259 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:31.294227    2572 kubelet.go:2658] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-76dff48d59-5bfgg"
May 26 11:27:31.294259 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:31.294259    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-76dff48d59-5bfgg"
May 26 11:27:31.294585 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:31.294572    2572 scope.go:117] "RemoveContainer" containerID="361a17845f710634401f0d559637fd64ef786ffebc338043bdcdbdc6003ca836"
May 26 11:27:31.294753 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:27:31.294737    2572 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-76dff48d59-5bfgg_openshift-console-operator(c6373342-dd07-4497-a4f3-978ea9613b6e)\"" pod="openshift-console-operator/console-operator-76dff48d59-5bfgg" podUID="c6373342-dd07-4497-a4f3-978ea9613b6e"
May 26 11:27:36.645380 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:36.645345    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ed80247a-4151-4422-b05e-a1764ec5d650-samples-operator-tls\") pod \"cluster-samples-operator-697587599-hls8q\" (UID: \"ed80247a-4151-4422-b05e-a1764ec5d650\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-hls8q"
May 26 11:27:36.647708 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:36.647684    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ed80247a-4151-4422-b05e-a1764ec5d650-samples-operator-tls\") pod \"cluster-samples-operator-697587599-hls8q\" (UID: \"ed80247a-4151-4422-b05e-a1764ec5d650\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-hls8q"
May 26 11:27:36.769608 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:36.769573    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-hls8q"
May 26 11:27:36.893969 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:36.893947    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-697587599-hls8q"]
May 26 11:27:37.263543 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:37.263457    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-hls8q" event={"ID":"ed80247a-4151-4422-b05e-a1764ec5d650","Type":"ContainerStarted","Data":"1abb2c172505f3a56c123c7aea942ec667e67393b190b95cae99407d6d291e83"}
May 26 11:27:37.754556 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:37.754522    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/97673f35-daf2-4eaf-b6cc-a119472f65c3-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6c75959455-xqrk8\" (UID: \"97673f35-daf2-4eaf-b6cc-a119472f65c3\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-xqrk8"
May 26 11:27:37.755001 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:27:37.754696    2572 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
May 26 11:27:37.755001 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:27:37.754775    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/97673f35-daf2-4eaf-b6cc-a119472f65c3-cluster-monitoring-operator-tls podName:97673f35-daf2-4eaf-b6cc-a119472f65c3 nodeName:}" failed. No retries permitted until 2026-05-26 11:27:53.754754591 +0000 UTC m=+196.656964905 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/97673f35-daf2-4eaf-b6cc-a119472f65c3-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6c75959455-xqrk8" (UID: "97673f35-daf2-4eaf-b6cc-a119472f65c3") : secret "cluster-monitoring-operator-tls" not found
May 26 11:27:39.270291 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:39.270258    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-hls8q" event={"ID":"ed80247a-4151-4422-b05e-a1764ec5d650","Type":"ContainerStarted","Data":"2bee2177397c5ac51aac2ff6586a402dafe76a0b2e60abad6d2085554c1a1027"}
May 26 11:27:39.270291 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:39.270293    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-hls8q" event={"ID":"ed80247a-4151-4422-b05e-a1764ec5d650","Type":"ContainerStarted","Data":"31124d8fac1770e671bba6fc27df8da69453aff32ab627dac98c207e27a2b678"}
May 26 11:27:39.292171 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:39.292128    2572 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-hls8q" podStartSLOduration=17.154056064 podStartE2EDuration="19.292113195s" podCreationTimestamp="2026-05-26 11:27:20 +0000 UTC" firstStartedPulling="2026-05-26 11:27:36.931799442 +0000 UTC m=+179.834009750" lastFinishedPulling="2026-05-26 11:27:39.069856568 +0000 UTC m=+181.972066881" observedRunningTime="2026-05-26 11:27:39.291163134 +0000 UTC m=+182.193373464" watchObservedRunningTime="2026-05-26 11:27:39.292113195 +0000 UTC m=+182.194323523"
May 26 11:27:46.671399 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:46.671367    2572 scope.go:117] "RemoveContainer" containerID="361a17845f710634401f0d559637fd64ef786ffebc338043bdcdbdc6003ca836"
May 26 11:27:47.289268 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:47.289236    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-5bfgg_c6373342-dd07-4497-a4f3-978ea9613b6e/console-operator/2.log"
May 26 11:27:47.289593 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:47.289577    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-5bfgg_c6373342-dd07-4497-a4f3-978ea9613b6e/console-operator/1.log"
May 26 11:27:47.289641 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:47.289610    2572 generic.go:358] "Generic (PLEG): container finished" podID="c6373342-dd07-4497-a4f3-978ea9613b6e" containerID="f44c90af71b6f623ba5c71c914aa721bae260713bd3952a240ac00d781181745" exitCode=255
May 26 11:27:47.289682 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:47.289672    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-76dff48d59-5bfgg" event={"ID":"c6373342-dd07-4497-a4f3-978ea9613b6e","Type":"ContainerDied","Data":"f44c90af71b6f623ba5c71c914aa721bae260713bd3952a240ac00d781181745"}
May 26 11:27:47.289734 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:47.289705    2572 scope.go:117] "RemoveContainer" containerID="361a17845f710634401f0d559637fd64ef786ffebc338043bdcdbdc6003ca836"
May 26 11:27:47.293918 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:47.290321    2572 scope.go:117] "RemoveContainer" containerID="f44c90af71b6f623ba5c71c914aa721bae260713bd3952a240ac00d781181745"
May 26 11:27:47.293918 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:27:47.290673    2572 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-76dff48d59-5bfgg_openshift-console-operator(c6373342-dd07-4497-a4f3-978ea9613b6e)\"" pod="openshift-console-operator/console-operator-76dff48d59-5bfgg" podUID="c6373342-dd07-4497-a4f3-978ea9613b6e"
May 26 11:27:48.070211 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:48.070155    2572 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-x7rdl"]
May 26 11:27:48.076167 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:48.076145    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-x7rdl"
May 26 11:27:48.078552 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:48.078532    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
May 26 11:27:48.079386 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:48.079370    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
May 26 11:27:48.079475 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:48.079392    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-lhjtv\""
May 26 11:27:48.092650 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:48.092627    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-x7rdl"]
May 26 11:27:48.237296 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:48.237260    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b2d3e1c9-98d6-4ec0-8906-cc73fab4e5c6-data-volume\") pod \"insights-runtime-extractor-x7rdl\" (UID: \"b2d3e1c9-98d6-4ec0-8906-cc73fab4e5c6\") " pod="openshift-insights/insights-runtime-extractor-x7rdl"
May 26 11:27:48.237472 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:48.237311    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b2d3e1c9-98d6-4ec0-8906-cc73fab4e5c6-crio-socket\") pod \"insights-runtime-extractor-x7rdl\" (UID: \"b2d3e1c9-98d6-4ec0-8906-cc73fab4e5c6\") " pod="openshift-insights/insights-runtime-extractor-x7rdl"
May 26 11:27:48.237472 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:48.237371    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b2d3e1c9-98d6-4ec0-8906-cc73fab4e5c6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-x7rdl\" (UID: \"b2d3e1c9-98d6-4ec0-8906-cc73fab4e5c6\") " pod="openshift-insights/insights-runtime-extractor-x7rdl"
May 26 11:27:48.237472 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:48.237396    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b2d3e1c9-98d6-4ec0-8906-cc73fab4e5c6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-x7rdl\" (UID: \"b2d3e1c9-98d6-4ec0-8906-cc73fab4e5c6\") " pod="openshift-insights/insights-runtime-extractor-x7rdl"
May 26 11:27:48.237472 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:48.237452    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8p6z\" (UniqueName: \"kubernetes.io/projected/b2d3e1c9-98d6-4ec0-8906-cc73fab4e5c6-kube-api-access-t8p6z\") pod \"insights-runtime-extractor-x7rdl\" (UID: \"b2d3e1c9-98d6-4ec0-8906-cc73fab4e5c6\") " pod="openshift-insights/insights-runtime-extractor-x7rdl"
May 26 11:27:48.293610 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:48.293582    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-5bfgg_c6373342-dd07-4497-a4f3-978ea9613b6e/console-operator/2.log"
May 26 11:27:48.338758 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:48.338690    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b2d3e1c9-98d6-4ec0-8906-cc73fab4e5c6-data-volume\") pod \"insights-runtime-extractor-x7rdl\" (UID: \"b2d3e1c9-98d6-4ec0-8906-cc73fab4e5c6\") " pod="openshift-insights/insights-runtime-extractor-x7rdl"
May 26 11:27:48.338758 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:48.338731    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b2d3e1c9-98d6-4ec0-8906-cc73fab4e5c6-crio-socket\") pod \"insights-runtime-extractor-x7rdl\" (UID: \"b2d3e1c9-98d6-4ec0-8906-cc73fab4e5c6\") " pod="openshift-insights/insights-runtime-extractor-x7rdl"
May 26 11:27:48.338894 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:48.338796    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b2d3e1c9-98d6-4ec0-8906-cc73fab4e5c6-crio-socket\") pod \"insights-runtime-extractor-x7rdl\" (UID: \"b2d3e1c9-98d6-4ec0-8906-cc73fab4e5c6\") " pod="openshift-insights/insights-runtime-extractor-x7rdl"
May 26 11:27:48.338894 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:48.338823    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b2d3e1c9-98d6-4ec0-8906-cc73fab4e5c6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-x7rdl\" (UID: \"b2d3e1c9-98d6-4ec0-8906-cc73fab4e5c6\") " pod="openshift-insights/insights-runtime-extractor-x7rdl"
May 26 11:27:48.338894 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:48.338842    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b2d3e1c9-98d6-4ec0-8906-cc73fab4e5c6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-x7rdl\" (UID: \"b2d3e1c9-98d6-4ec0-8906-cc73fab4e5c6\") " pod="openshift-insights/insights-runtime-extractor-x7rdl"
May 26 11:27:48.338894 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:48.338881    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-t8p6z\" (UniqueName: \"kubernetes.io/projected/b2d3e1c9-98d6-4ec0-8906-cc73fab4e5c6-kube-api-access-t8p6z\") pod \"insights-runtime-extractor-x7rdl\" (UID: \"b2d3e1c9-98d6-4ec0-8906-cc73fab4e5c6\") " pod="openshift-insights/insights-runtime-extractor-x7rdl"
May 26 11:27:48.339087 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:48.339065    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b2d3e1c9-98d6-4ec0-8906-cc73fab4e5c6-data-volume\") pod \"insights-runtime-extractor-x7rdl\" (UID: \"b2d3e1c9-98d6-4ec0-8906-cc73fab4e5c6\") " pod="openshift-insights/insights-runtime-extractor-x7rdl"
May 26 11:27:48.339344 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:48.339328    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b2d3e1c9-98d6-4ec0-8906-cc73fab4e5c6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-x7rdl\" (UID: \"b2d3e1c9-98d6-4ec0-8906-cc73fab4e5c6\") " pod="openshift-insights/insights-runtime-extractor-x7rdl"
May 26 11:27:48.341254 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:48.341223    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b2d3e1c9-98d6-4ec0-8906-cc73fab4e5c6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-x7rdl\" (UID: \"b2d3e1c9-98d6-4ec0-8906-cc73fab4e5c6\") " pod="openshift-insights/insights-runtime-extractor-x7rdl"
May 26 11:27:48.362051 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:48.362026    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8p6z\" (UniqueName: \"kubernetes.io/projected/b2d3e1c9-98d6-4ec0-8906-cc73fab4e5c6-kube-api-access-t8p6z\") pod \"insights-runtime-extractor-x7rdl\" (UID: \"b2d3e1c9-98d6-4ec0-8906-cc73fab4e5c6\") " pod="openshift-insights/insights-runtime-extractor-x7rdl"
May 26 11:27:48.384889 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:48.384865    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-x7rdl"
May 26 11:27:48.509481 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:48.509450    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-x7rdl"]
May 26 11:27:48.512418 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:27:48.512389    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2d3e1c9_98d6_4ec0_8906_cc73fab4e5c6.slice/crio-94455aba37525cc2d62999939130f594a1a0b6ca8478541ee45ddd1e5b8a2f11 WatchSource:0}: Error finding container 94455aba37525cc2d62999939130f594a1a0b6ca8478541ee45ddd1e5b8a2f11: Status 404 returned error can't find the container with id 94455aba37525cc2d62999939130f594a1a0b6ca8478541ee45ddd1e5b8a2f11
May 26 11:27:49.297131 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:49.297099    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-x7rdl" event={"ID":"b2d3e1c9-98d6-4ec0-8906-cc73fab4e5c6","Type":"ContainerStarted","Data":"1580ab7ae9aa506f55a2f316afabbd4d6b6b766478d3a04cb2916ccc9f4d32b8"}
May 26 11:27:49.297131 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:49.297133    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-x7rdl" event={"ID":"b2d3e1c9-98d6-4ec0-8906-cc73fab4e5c6","Type":"ContainerStarted","Data":"c3de5a7242afc01dea46c473372b299a37b8706ec100b5c175dae97d9b8828a0"}
May 26 11:27:49.297525 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:49.297144    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-x7rdl" event={"ID":"b2d3e1c9-98d6-4ec0-8906-cc73fab4e5c6","Type":"ContainerStarted","Data":"94455aba37525cc2d62999939130f594a1a0b6ca8478541ee45ddd1e5b8a2f11"}
May 26 11:27:51.293911 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:51.293876    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-76dff48d59-5bfgg"
May 26 11:27:51.294429 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:51.293977    2572 kubelet.go:2658] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-76dff48d59-5bfgg"
May 26 11:27:51.294429 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:51.294214    2572 scope.go:117] "RemoveContainer" containerID="f44c90af71b6f623ba5c71c914aa721bae260713bd3952a240ac00d781181745"
May 26 11:27:51.294429 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:27:51.294366    2572 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-76dff48d59-5bfgg_openshift-console-operator(c6373342-dd07-4497-a4f3-978ea9613b6e)\"" pod="openshift-console-operator/console-operator-76dff48d59-5bfgg" podUID="c6373342-dd07-4497-a4f3-978ea9613b6e"
May 26 11:27:51.303792 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:51.303767    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-x7rdl" event={"ID":"b2d3e1c9-98d6-4ec0-8906-cc73fab4e5c6","Type":"ContainerStarted","Data":"d2a4cc9cb8becef324bc8887638d47d06d8c51ceaefc2b8173da67f8367c18fc"}
May 26 11:27:51.304043 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:51.304029    2572 scope.go:117] "RemoveContainer" containerID="f44c90af71b6f623ba5c71c914aa721bae260713bd3952a240ac00d781181745"
May 26 11:27:51.304214 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:27:51.304176    2572 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-76dff48d59-5bfgg_openshift-console-operator(c6373342-dd07-4497-a4f3-978ea9613b6e)\"" pod="openshift-console-operator/console-operator-76dff48d59-5bfgg" podUID="c6373342-dd07-4497-a4f3-978ea9613b6e"
May 26 11:27:51.341880 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:51.341818    2572 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-x7rdl" podStartSLOduration=1.537766025 podStartE2EDuration="3.341803672s" podCreationTimestamp="2026-05-26 11:27:48 +0000 UTC" firstStartedPulling="2026-05-26 11:27:48.567283136 +0000 UTC m=+191.469493447" lastFinishedPulling="2026-05-26 11:27:50.37132057 +0000 UTC m=+193.273531094" observedRunningTime="2026-05-26 11:27:51.34137651 +0000 UTC m=+194.243586839" watchObservedRunningTime="2026-05-26 11:27:51.341803672 +0000 UTC m=+194.244014002"
May 26 11:27:53.783874 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:53.783828    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/97673f35-daf2-4eaf-b6cc-a119472f65c3-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6c75959455-xqrk8\" (UID: \"97673f35-daf2-4eaf-b6cc-a119472f65c3\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-xqrk8"
May 26 11:27:53.786109 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:53.786086    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/97673f35-daf2-4eaf-b6cc-a119472f65c3-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6c75959455-xqrk8\" (UID: \"97673f35-daf2-4eaf-b6cc-a119472f65c3\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-xqrk8"
May 26 11:27:53.990724 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:53.990695    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-g8gk4\""
May 26 11:27:53.997900 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:53.997884    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-xqrk8"
May 26 11:27:54.119234 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:54.119199    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6c75959455-xqrk8"]
May 26 11:27:54.123736 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:27:54.123710    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97673f35_daf2_4eaf_b6cc_a119472f65c3.slice/crio-c7d0ffaf143a7d5d9f01585693f42623967e8f19cbbdd8130e68bfea725a99df WatchSource:0}: Error finding container c7d0ffaf143a7d5d9f01585693f42623967e8f19cbbdd8130e68bfea725a99df: Status 404 returned error can't find the container with id c7d0ffaf143a7d5d9f01585693f42623967e8f19cbbdd8130e68bfea725a99df
May 26 11:27:54.311321 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:54.311236    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-xqrk8" event={"ID":"97673f35-daf2-4eaf-b6cc-a119472f65c3","Type":"ContainerStarted","Data":"c7d0ffaf143a7d5d9f01585693f42623967e8f19cbbdd8130e68bfea725a99df"}
May 26 11:27:56.318804 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:56.318761    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-xqrk8" event={"ID":"97673f35-daf2-4eaf-b6cc-a119472f65c3","Type":"ContainerStarted","Data":"0823fc8f5c09a1747cef0af8044de6fb70aceab5ec2009623540c7dbb4e701d3"}
May 26 11:27:56.340699 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:56.340645    2572 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-xqrk8" podStartSLOduration=34.360080623 podStartE2EDuration="36.340627834s" podCreationTimestamp="2026-05-26 11:27:20 +0000 UTC" firstStartedPulling="2026-05-26 11:27:54.125519604 +0000 UTC m=+197.027729911" lastFinishedPulling="2026-05-26 11:27:56.106066816 +0000 UTC m=+199.008277122" observedRunningTime="2026-05-26 11:27:56.339794464 +0000 UTC m=+199.242004794" watchObservedRunningTime="2026-05-26 11:27:56.340627834 +0000 UTC m=+199.242838164"
May 26 11:27:56.714541 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:56.714510    2572 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-zsds2"]
May 26 11:27:56.716522 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:56.716507    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-zsds2"
May 26 11:27:56.719350 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:56.719333    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-fkrgj\""
May 26 11:27:56.720129 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:56.720114    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
May 26 11:27:56.729738 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:56.729715    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-zsds2"]
May 26 11:27:56.810386 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:56.810349    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/1ff04742-c904-42ce-b625-308ef359b822-tls-certificates\") pod \"prometheus-operator-admission-webhook-6c4bbc67b7-zsds2\" (UID: \"1ff04742-c904-42ce-b625-308ef359b822\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-zsds2"
May 26 11:27:56.911630 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:56.911584    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/1ff04742-c904-42ce-b625-308ef359b822-tls-certificates\") pod \"prometheus-operator-admission-webhook-6c4bbc67b7-zsds2\" (UID: \"1ff04742-c904-42ce-b625-308ef359b822\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-zsds2"
May 26 11:27:56.911810 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:27:56.911744    2572 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found
May 26 11:27:56.911856 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:27:56.911813    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1ff04742-c904-42ce-b625-308ef359b822-tls-certificates podName:1ff04742-c904-42ce-b625-308ef359b822 nodeName:}" failed. No retries permitted until 2026-05-26 11:27:57.411796807 +0000 UTC m=+200.314007119 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/1ff04742-c904-42ce-b625-308ef359b822-tls-certificates") pod "prometheus-operator-admission-webhook-6c4bbc67b7-zsds2" (UID: "1ff04742-c904-42ce-b625-308ef359b822") : secret "prometheus-operator-admission-webhook-tls" not found
May 26 11:27:57.415650 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:57.415607    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/1ff04742-c904-42ce-b625-308ef359b822-tls-certificates\") pod \"prometheus-operator-admission-webhook-6c4bbc67b7-zsds2\" (UID: \"1ff04742-c904-42ce-b625-308ef359b822\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-zsds2"
May 26 11:27:57.418034 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:57.418011    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/1ff04742-c904-42ce-b625-308ef359b822-tls-certificates\") pod \"prometheus-operator-admission-webhook-6c4bbc67b7-zsds2\" (UID: \"1ff04742-c904-42ce-b625-308ef359b822\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-zsds2"
May 26 11:27:57.624799 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:57.624752    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-zsds2"
May 26 11:27:57.751305 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:57.751213    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-zsds2"]
May 26 11:27:57.753619 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:27:57.753588    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ff04742_c904_42ce_b625_308ef359b822.slice/crio-3121ee9241941fb000752f29e213515523f4629de12ffc73312d46d01541663d WatchSource:0}: Error finding container 3121ee9241941fb000752f29e213515523f4629de12ffc73312d46d01541663d: Status 404 returned error can't find the container with id 3121ee9241941fb000752f29e213515523f4629de12ffc73312d46d01541663d
May 26 11:27:58.326713 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:58.326661    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-zsds2" event={"ID":"1ff04742-c904-42ce-b625-308ef359b822","Type":"ContainerStarted","Data":"3121ee9241941fb000752f29e213515523f4629de12ffc73312d46d01541663d"}
May 26 11:27:59.330368 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:59.330331    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-zsds2" event={"ID":"1ff04742-c904-42ce-b625-308ef359b822","Type":"ContainerStarted","Data":"afdf2cc6266364d30b4fd3de522e49faea2a0c017880921178fe80d470a01872"}
May 26 11:27:59.330840 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:59.330568    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-zsds2"
May 26 11:27:59.335152 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:59.335128    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-zsds2"
May 26 11:27:59.348382 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:59.348342    2572 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-zsds2" podStartSLOduration=2.51666395 podStartE2EDuration="3.348330222s" podCreationTimestamp="2026-05-26 11:27:56 +0000 UTC" firstStartedPulling="2026-05-26 11:27:57.75544928 +0000 UTC m=+200.657659590" lastFinishedPulling="2026-05-26 11:27:58.587115554 +0000 UTC m=+201.489325862" observedRunningTime="2026-05-26 11:27:59.346847035 +0000 UTC m=+202.249057363" watchObservedRunningTime="2026-05-26 11:27:59.348330222 +0000 UTC m=+202.250540551"
May 26 11:27:59.789227 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:59.789178    2572 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-7779bbdd78-cnpnj"]
May 26 11:27:59.791168 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:59.791153    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-7779bbdd78-cnpnj"
May 26 11:27:59.793504 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:59.793478    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
May 26 11:27:59.794834 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:59.794813    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-w8ws6\""
May 26 11:27:59.794947 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:59.794873    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
May 26 11:27:59.794947 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:59.794880    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
May 26 11:27:59.804924 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:59.804901    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-7779bbdd78-cnpnj"]
May 26 11:27:59.836616 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:59.836590    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/facd2dc8-401d-4a76-98e0-d7fa73b350b1-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7779bbdd78-cnpnj\" (UID: \"facd2dc8-401d-4a76-98e0-d7fa73b350b1\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-cnpnj"
May 26 11:27:59.836710 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:59.836662    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/facd2dc8-401d-4a76-98e0-d7fa73b350b1-prometheus-operator-tls\") pod \"prometheus-operator-7779bbdd78-cnpnj\" (UID: \"facd2dc8-401d-4a76-98e0-d7fa73b350b1\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-cnpnj"
May 26 11:27:59.836710 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:59.836684    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/facd2dc8-401d-4a76-98e0-d7fa73b350b1-metrics-client-ca\") pod \"prometheus-operator-7779bbdd78-cnpnj\" (UID: \"facd2dc8-401d-4a76-98e0-d7fa73b350b1\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-cnpnj"
May 26 11:27:59.836781 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:59.836720    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c69xt\" (UniqueName: \"kubernetes.io/projected/facd2dc8-401d-4a76-98e0-d7fa73b350b1-kube-api-access-c69xt\") pod \"prometheus-operator-7779bbdd78-cnpnj\" (UID: \"facd2dc8-401d-4a76-98e0-d7fa73b350b1\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-cnpnj"
May 26 11:27:59.937652 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:59.937603    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/facd2dc8-401d-4a76-98e0-d7fa73b350b1-prometheus-operator-tls\") pod \"prometheus-operator-7779bbdd78-cnpnj\" (UID: \"facd2dc8-401d-4a76-98e0-d7fa73b350b1\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-cnpnj"
May 26 11:27:59.937652 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:59.937657    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/facd2dc8-401d-4a76-98e0-d7fa73b350b1-metrics-client-ca\") pod \"prometheus-operator-7779bbdd78-cnpnj\" (UID: \"facd2dc8-401d-4a76-98e0-d7fa73b350b1\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-cnpnj"
May 26 11:27:59.937825 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:59.937696    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-c69xt\" (UniqueName: \"kubernetes.io/projected/facd2dc8-401d-4a76-98e0-d7fa73b350b1-kube-api-access-c69xt\") pod \"prometheus-operator-7779bbdd78-cnpnj\" (UID: \"facd2dc8-401d-4a76-98e0-d7fa73b350b1\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-cnpnj"
May 26 11:27:59.937825 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:59.937737    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/facd2dc8-401d-4a76-98e0-d7fa73b350b1-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7779bbdd78-cnpnj\" (UID: \"facd2dc8-401d-4a76-98e0-d7fa73b350b1\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-cnpnj"
May 26 11:27:59.938475 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:59.938454    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/facd2dc8-401d-4a76-98e0-d7fa73b350b1-metrics-client-ca\") pod \"prometheus-operator-7779bbdd78-cnpnj\" (UID: \"facd2dc8-401d-4a76-98e0-d7fa73b350b1\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-cnpnj"
May 26 11:27:59.940046 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:59.940018    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/facd2dc8-401d-4a76-98e0-d7fa73b350b1-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7779bbdd78-cnpnj\" (UID: \"facd2dc8-401d-4a76-98e0-d7fa73b350b1\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-cnpnj"
May 26 11:27:59.940046 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:59.940032    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/facd2dc8-401d-4a76-98e0-d7fa73b350b1-prometheus-operator-tls\") pod \"prometheus-operator-7779bbdd78-cnpnj\" (UID: \"facd2dc8-401d-4a76-98e0-d7fa73b350b1\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-cnpnj"
May 26 11:27:59.946261 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:27:59.946239    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c69xt\" (UniqueName: \"kubernetes.io/projected/facd2dc8-401d-4a76-98e0-d7fa73b350b1-kube-api-access-c69xt\") pod \"prometheus-operator-7779bbdd78-cnpnj\" (UID: \"facd2dc8-401d-4a76-98e0-d7fa73b350b1\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-cnpnj"
May 26 11:28:00.100176 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:00.100101    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-7779bbdd78-cnpnj"
May 26 11:28:00.226001 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:00.225966    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-7779bbdd78-cnpnj"]
May 26 11:28:00.228862 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:28:00.228825    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfacd2dc8_401d_4a76_98e0_d7fa73b350b1.slice/crio-6fad2827688900e7906a734aabebe3fcacd38651edcc82d0452549d16453a2e4 WatchSource:0}: Error finding container 6fad2827688900e7906a734aabebe3fcacd38651edcc82d0452549d16453a2e4: Status 404 returned error can't find the container with id 6fad2827688900e7906a734aabebe3fcacd38651edcc82d0452549d16453a2e4
May 26 11:28:00.334243 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:00.334210    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7779bbdd78-cnpnj" event={"ID":"facd2dc8-401d-4a76-98e0-d7fa73b350b1","Type":"ContainerStarted","Data":"6fad2827688900e7906a734aabebe3fcacd38651edcc82d0452549d16453a2e4"}
May 26 11:28:01.338742 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:01.338705    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7779bbdd78-cnpnj" event={"ID":"facd2dc8-401d-4a76-98e0-d7fa73b350b1","Type":"ContainerStarted","Data":"2985001d3476e1e0993a445723f3bcd1de3e31961130569332a17fb1fb28ba5b"}
May 26 11:28:02.345371 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:02.345334    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7779bbdd78-cnpnj" event={"ID":"facd2dc8-401d-4a76-98e0-d7fa73b350b1","Type":"ContainerStarted","Data":"3bfb19cd1401a4b8b7bf440e03903b6c4bb814adad3025d3342dc4a7dcd7843c"}
May 26 11:28:02.367687 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:02.367637    2572 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-7779bbdd78-cnpnj" podStartSLOduration=2.31989069 podStartE2EDuration="3.367622529s" podCreationTimestamp="2026-05-26 11:27:59 +0000 UTC" firstStartedPulling="2026-05-26 11:28:00.230694251 +0000 UTC m=+203.132904561" lastFinishedPulling="2026-05-26 11:28:01.278426089 +0000 UTC m=+204.180636400" observedRunningTime="2026-05-26 11:28:02.367118953 +0000 UTC m=+205.269329282" watchObservedRunningTime="2026-05-26 11:28:02.367622529 +0000 UTC m=+205.269832896"
May 26 11:28:04.191162 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:04.191125    2572 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-9c959559f-nwmwj"]
May 26 11:28:04.193334 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:04.193316    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-9c959559f-nwmwj"
May 26 11:28:04.195811 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:04.195790    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
May 26 11:28:04.196069 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:04.196053    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
May 26 11:28:04.201875 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:04.201855    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-p5xwg\""
May 26 11:28:04.214717 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:04.214694    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-9c959559f-nwmwj"]
May 26 11:28:04.217205 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:04.217166    2572 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-54fd6bc5c4-88ht6"]
May 26 11:28:04.219296 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:04.219279    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-88ht6"
May 26 11:28:04.223307 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:04.223285    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
May 26 11:28:04.224325 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:04.224305    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
May 26 11:28:04.224425 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:04.224305    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
May 26 11:28:04.224425 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:04.224366    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-j7kw4\""
May 26 11:28:04.233851 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:04.233826    2572 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-f787b"]
May 26 11:28:04.235846 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:04.235829    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-f787b"
May 26 11:28:04.239070 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:04.239050    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
May 26 11:28:04.239170 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:04.239078    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
May 26 11:28:04.239265 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:04.239224    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
May 26 11:28:04.239351 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:04.239332    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-5tkgs\""
May 26 11:28:04.245063 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:04.245042    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-54fd6bc5c4-88ht6"]
May 26 11:28:04.272599 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:04.272571    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5ba9e31e-1334-452d-92d6-4540727e1831-sys\") pod \"node-exporter-f787b\" (UID: \"5ba9e31e-1334-452d-92d6-4540727e1831\") " pod="openshift-monitoring/node-exporter-f787b"
May 26 11:28:04.272743 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:04.272615    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dfe13380-0f06-46a6-b295-86f0bc1694a4-metrics-client-ca\") pod \"kube-state-metrics-54fd6bc5c4-88ht6\" (UID: \"dfe13380-0f06-46a6-b295-86f0bc1694a4\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-88ht6"
May 26 11:28:04.272743 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:04.272635    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/dfe13380-0f06-46a6-b295-86f0bc1694a4-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-54fd6bc5c4-88ht6\" (UID: \"dfe13380-0f06-46a6-b295-86f0bc1694a4\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-88ht6"
May 26 11:28:04.272743 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:04.272651    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5ba9e31e-1334-452d-92d6-4540727e1831-node-exporter-tls\") pod \"node-exporter-f787b\" (UID: \"5ba9e31e-1334-452d-92d6-4540727e1831\") " pod="openshift-monitoring/node-exporter-f787b"
May 26 11:28:04.272743 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:04.272667    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/5ba9e31e-1334-452d-92d6-4540727e1831-node-exporter-accelerators-collector-config\") pod \"node-exporter-f787b\" (UID: \"5ba9e31e-1334-452d-92d6-4540727e1831\") " pod="openshift-monitoring/node-exporter-f787b"
May 26 11:28:04.272743 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:04.272688    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/dfe13380-0f06-46a6-b295-86f0bc1694a4-volume-directive-shadow\") pod \"kube-state-metrics-54fd6bc5c4-88ht6\" (UID: \"dfe13380-0f06-46a6-b295-86f0bc1694a4\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-88ht6"
May 26 11:28:04.272743 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:04.272704    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l85fs\" (UniqueName: \"kubernetes.io/projected/dfe13380-0f06-46a6-b295-86f0bc1694a4-kube-api-access-l85fs\") pod \"kube-state-metrics-54fd6bc5c4-88ht6\" (UID: \"dfe13380-0f06-46a6-b295-86f0bc1694a4\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-88ht6"
May 26 11:28:04.272991 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:04.272767    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/922726b6-be61-448e-b8ab-6e30b7ccfcbf-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-9c959559f-nwmwj\" (UID: \"922726b6-be61-448e-b8ab-6e30b7ccfcbf\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-nwmwj"
May 26 11:28:04.272991 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:04.272795    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wnxmg\" (UniqueName: \"kubernetes.io/projected/922726b6-be61-448e-b8ab-6e30b7ccfcbf-kube-api-access-wnxmg\") pod \"openshift-state-metrics-9c959559f-nwmwj\" (UID: \"922726b6-be61-448e-b8ab-6e30b7ccfcbf\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-nwmwj"
May 26 11:28:04.272991 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:04.272820    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sb7d2\" (UniqueName: \"kubernetes.io/projected/5ba9e31e-1334-452d-92d6-4540727e1831-kube-api-access-sb7d2\") pod \"node-exporter-f787b\" (UID: \"5ba9e31e-1334-452d-92d6-4540727e1831\") " pod="openshift-monitoring/node-exporter-f787b"
May 26 11:28:04.272991 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:04.272869    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5ba9e31e-1334-452d-92d6-4540727e1831-metrics-client-ca\") pod \"node-exporter-f787b\" (UID: \"5ba9e31e-1334-452d-92d6-4540727e1831\") " pod="openshift-monitoring/node-exporter-f787b"
May 26 11:28:04.272991 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:04.272907    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dfe13380-0f06-46a6-b295-86f0bc1694a4-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-54fd6bc5c4-88ht6\" (UID: \"dfe13380-0f06-46a6-b295-86f0bc1694a4\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-88ht6"
May 26 11:28:04.272991 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:04.272954    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/5ba9e31e-1334-452d-92d6-4540727e1831-node-exporter-textfile\") pod \"node-exporter-f787b\" (UID: \"5ba9e31e-1334-452d-92d6-4540727e1831\") " pod="openshift-monitoring/node-exporter-f787b"
May 26 11:28:04.273297 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:04.273022    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/dfe13380-0f06-46a6-b295-86f0bc1694a4-kube-state-metrics-tls\") pod \"kube-state-metrics-54fd6bc5c4-88ht6\" (UID: \"dfe13380-0f06-46a6-b295-86f0bc1694a4\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-88ht6"
May 26 11:28:04.273297 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:04.273052    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5ba9e31e-1334-452d-92d6-4540727e1831-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-f787b\" (UID: \"5ba9e31e-1334-452d-92d6-4540727e1831\") " pod="openshift-monitoring/node-exporter-f787b"
May 26 11:28:04.273297 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:04.273088    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/922726b6-be61-448e-b8ab-6e30b7ccfcbf-metrics-client-ca\") pod \"openshift-state-metrics-9c959559f-nwmwj\" (UID: \"922726b6-be61-448e-b8ab-6e30b7ccfcbf\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-nwmwj"
May 26 11:28:04.273297 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:04.273138    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/5ba9e31e-1334-452d-92d6-4540727e1831-node-exporter-wtmp\") pod \"node-exporter-f787b\" (UID: \"5ba9e31e-1334-452d-92d6-4540727e1831\") " pod="openshift-monitoring/node-exporter-f787b"
May 26 11:28:04.273297 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:04.273165    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/922726b6-be61-448e-b8ab-6e30b7ccfcbf-openshift-state-metrics-tls\") pod \"openshift-state-metrics-9c959559f-nwmwj\" (UID: \"922726b6-be61-448e-b8ab-6e30b7ccfcbf\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-nwmwj"
May 26 11:28:04.273297 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:04.273208    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/5ba9e31e-1334-452d-92d6-4540727e1831-root\") pod \"node-exporter-f787b\" (UID: \"5ba9e31e-1334-452d-92d6-4540727e1831\") " pod="openshift-monitoring/node-exporter-f787b"
May 26 11:28:04.374445 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:04.374412    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/dfe13380-0f06-46a6-b295-86f0bc1694a4-kube-state-metrics-tls\") pod \"kube-state-metrics-54fd6bc5c4-88ht6\" (UID: \"dfe13380-0f06-46a6-b295-86f0bc1694a4\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-88ht6"
May 26 11:28:04.374445 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:04.374447    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5ba9e31e-1334-452d-92d6-4540727e1831-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-f787b\" (UID: \"5ba9e31e-1334-452d-92d6-4540727e1831\") " pod="openshift-monitoring/node-exporter-f787b"
May 26 11:28:04.374704 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:04.374469    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/922726b6-be61-448e-b8ab-6e30b7ccfcbf-metrics-client-ca\") pod \"openshift-state-metrics-9c959559f-nwmwj\" (UID: \"922726b6-be61-448e-b8ab-6e30b7ccfcbf\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-nwmwj"
May 26 11:28:04.374704 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:04.374589    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/5ba9e31e-1334-452d-92d6-4540727e1831-node-exporter-wtmp\") pod \"node-exporter-f787b\" (UID: \"5ba9e31e-1334-452d-92d6-4540727e1831\") " pod="openshift-monitoring/node-exporter-f787b"
May 26 11:28:04.374704 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:04.374622    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/922726b6-be61-448e-b8ab-6e30b7ccfcbf-openshift-state-metrics-tls\") pod \"openshift-state-metrics-9c959559f-nwmwj\" (UID: \"922726b6-be61-448e-b8ab-6e30b7ccfcbf\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-nwmwj"
May 26 11:28:04.374704 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:04.374646    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/5ba9e31e-1334-452d-92d6-4540727e1831-root\") pod \"node-exporter-f787b\" (UID: \"5ba9e31e-1334-452d-92d6-4540727e1831\") " pod="openshift-monitoring/node-exporter-f787b"
May 26 11:28:04.374704 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:04.374696    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/5ba9e31e-1334-452d-92d6-4540727e1831-root\") pod \"node-exporter-f787b\" (UID: \"5ba9e31e-1334-452d-92d6-4540727e1831\") " pod="openshift-monitoring/node-exporter-f787b"
May 26 11:28:04.374952 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:04.374744    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5ba9e31e-1334-452d-92d6-4540727e1831-sys\") pod \"node-exporter-f787b\" (UID: \"5ba9e31e-1334-452d-92d6-4540727e1831\") " pod="openshift-monitoring/node-exporter-f787b"
May 26 11:28:04.374952 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:04.374789    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dfe13380-0f06-46a6-b295-86f0bc1694a4-metrics-client-ca\") pod \"kube-state-metrics-54fd6bc5c4-88ht6\" (UID: \"dfe13380-0f06-46a6-b295-86f0bc1694a4\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-88ht6"
May 26 11:28:04.374952 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:04.374804    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5ba9e31e-1334-452d-92d6-4540727e1831-sys\") pod \"node-exporter-f787b\" (UID: \"5ba9e31e-1334-452d-92d6-4540727e1831\") " pod="openshift-monitoring/node-exporter-f787b"
May 26 11:28:04.374952 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:04.374751    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/5ba9e31e-1334-452d-92d6-4540727e1831-node-exporter-wtmp\") pod \"node-exporter-f787b\" (UID: \"5ba9e31e-1334-452d-92d6-4540727e1831\") " pod="openshift-monitoring/node-exporter-f787b"
May 26 11:28:04.374952 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:04.374819    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/dfe13380-0f06-46a6-b295-86f0bc1694a4-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-54fd6bc5c4-88ht6\" (UID: \"dfe13380-0f06-46a6-b295-86f0bc1694a4\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-88ht6"
May 26 11:28:04.374952 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:04.374849    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5ba9e31e-1334-452d-92d6-4540727e1831-node-exporter-tls\") pod \"node-exporter-f787b\" (UID: \"5ba9e31e-1334-452d-92d6-4540727e1831\") " pod="openshift-monitoring/node-exporter-f787b"
May 26 11:28:04.374952 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:04.374875    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/5ba9e31e-1334-452d-92d6-4540727e1831-node-exporter-accelerators-collector-config\") pod \"node-exporter-f787b\" (UID: \"5ba9e31e-1334-452d-92d6-4540727e1831\") " pod="openshift-monitoring/node-exporter-f787b"
May 26 11:28:04.374952 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:04.374916    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/dfe13380-0f06-46a6-b295-86f0bc1694a4-volume-directive-shadow\") pod \"kube-state-metrics-54fd6bc5c4-88ht6\" (UID: \"dfe13380-0f06-46a6-b295-86f0bc1694a4\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-88ht6"
May 26 11:28:04.374952 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:04.374942    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-l85fs\" (UniqueName: \"kubernetes.io/projected/dfe13380-0f06-46a6-b295-86f0bc1694a4-kube-api-access-l85fs\") pod \"kube-state-metrics-54fd6bc5c4-88ht6\" (UID: \"dfe13380-0f06-46a6-b295-86f0bc1694a4\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-88ht6"
May 26 11:28:04.375389 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:04.374980    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/922726b6-be61-448e-b8ab-6e30b7ccfcbf-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-9c959559f-nwmwj\" (UID: \"922726b6-be61-448e-b8ab-6e30b7ccfcbf\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-nwmwj"
May 26 11:28:04.375389 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:28:04.374995    2572 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
May 26 11:28:04.375389 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:04.375009    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-wnxmg\" (UniqueName: \"kubernetes.io/projected/922726b6-be61-448e-b8ab-6e30b7ccfcbf-kube-api-access-wnxmg\") pod \"openshift-state-metrics-9c959559f-nwmwj\" (UID: \"922726b6-be61-448e-b8ab-6e30b7ccfcbf\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-nwmwj"
May 26 11:28:04.375389 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:04.375042    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-sb7d2\" (UniqueName: \"kubernetes.io/projected/5ba9e31e-1334-452d-92d6-4540727e1831-kube-api-access-sb7d2\") pod \"node-exporter-f787b\" (UID: \"5ba9e31e-1334-452d-92d6-4540727e1831\") " pod="openshift-monitoring/node-exporter-f787b"
May 26 11:28:04.375389 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:28:04.375078    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5ba9e31e-1334-452d-92d6-4540727e1831-node-exporter-tls podName:5ba9e31e-1334-452d-92d6-4540727e1831 nodeName:}" failed. No retries permitted until 2026-05-26 11:28:04.875057075 +0000 UTC m=+207.777267398 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/5ba9e31e-1334-452d-92d6-4540727e1831-node-exporter-tls") pod "node-exporter-f787b" (UID: "5ba9e31e-1334-452d-92d6-4540727e1831") : secret "node-exporter-tls" not found
May 26 11:28:04.375389 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:04.375134    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5ba9e31e-1334-452d-92d6-4540727e1831-metrics-client-ca\") pod \"node-exporter-f787b\" (UID: \"5ba9e31e-1334-452d-92d6-4540727e1831\") " pod="openshift-monitoring/node-exporter-f787b"
May 26 11:28:04.375389 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:04.375212    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dfe13380-0f06-46a6-b295-86f0bc1694a4-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-54fd6bc5c4-88ht6\" (UID: \"dfe13380-0f06-46a6-b295-86f0bc1694a4\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-88ht6"
May 26 11:28:04.375389 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:04.375246    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/5ba9e31e-1334-452d-92d6-4540727e1831-node-exporter-textfile\") pod \"node-exporter-f787b\" (UID: \"5ba9e31e-1334-452d-92d6-4540727e1831\") " pod="openshift-monitoring/node-exporter-f787b"
May 26 11:28:04.375780 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:04.375519    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dfe13380-0f06-46a6-b295-86f0bc1694a4-metrics-client-ca\") pod \"kube-state-metrics-54fd6bc5c4-88ht6\" (UID: \"dfe13380-0f06-46a6-b295-86f0bc1694a4\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-88ht6"
May 26 11:28:04.375780 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:04.375542    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/5ba9e31e-1334-452d-92d6-4540727e1831-node-exporter-textfile\") pod \"node-exporter-f787b\" (UID: \"5ba9e31e-1334-452d-92d6-4540727e1831\") " pod="openshift-monitoring/node-exporter-f787b"
May 26 11:28:04.375780 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:04.375214    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/922726b6-be61-448e-b8ab-6e30b7ccfcbf-metrics-client-ca\") pod \"openshift-state-metrics-9c959559f-nwmwj\" (UID: \"922726b6-be61-448e-b8ab-6e30b7ccfcbf\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-nwmwj"
May 26 11:28:04.375921 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:04.375839    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/dfe13380-0f06-46a6-b295-86f0bc1694a4-volume-directive-shadow\") pod \"kube-state-metrics-54fd6bc5c4-88ht6\" (UID: \"dfe13380-0f06-46a6-b295-86f0bc1694a4\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-88ht6"
May 26 11:28:04.375921 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:04.375874    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/5ba9e31e-1334-452d-92d6-4540727e1831-node-exporter-accelerators-collector-config\") pod \"node-exporter-f787b\" (UID: \"5ba9e31e-1334-452d-92d6-4540727e1831\") " pod="openshift-monitoring/node-exporter-f787b"
May 26 11:28:04.376039 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:04.376016    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5ba9e31e-1334-452d-92d6-4540727e1831-metrics-client-ca\") pod \"node-exporter-f787b\" (UID: \"5ba9e31e-1334-452d-92d6-4540727e1831\") " pod="openshift-monitoring/node-exporter-f787b"
May 26 11:28:04.376619 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:04.376597    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/dfe13380-0f06-46a6-b295-86f0bc1694a4-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-54fd6bc5c4-88ht6\" (UID: \"dfe13380-0f06-46a6-b295-86f0bc1694a4\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-88ht6"
May 26 11:28:04.377223 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:04.377181    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5ba9e31e-1334-452d-92d6-4540727e1831-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-f787b\" (UID: \"5ba9e31e-1334-452d-92d6-4540727e1831\") " pod="openshift-monitoring/node-exporter-f787b"
May 26 11:28:04.377570 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:04.377550    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/dfe13380-0f06-46a6-b295-86f0bc1694a4-kube-state-metrics-tls\") pod \"kube-state-metrics-54fd6bc5c4-88ht6\" (UID: \"dfe13380-0f06-46a6-b295-86f0bc1694a4\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-88ht6"
May 26 11:28:04.377610 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:04.377550    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/922726b6-be61-448e-b8ab-6e30b7ccfcbf-openshift-state-metrics-tls\") pod \"openshift-state-metrics-9c959559f-nwmwj\" (UID: \"922726b6-be61-448e-b8ab-6e30b7ccfcbf\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-nwmwj"
May 26 11:28:04.378288 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:04.378272    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dfe13380-0f06-46a6-b295-86f0bc1694a4-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-54fd6bc5c4-88ht6\" (UID: \"dfe13380-0f06-46a6-b295-86f0bc1694a4\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-88ht6"
May 26 11:28:04.378387 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:04.378368    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/922726b6-be61-448e-b8ab-6e30b7ccfcbf-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-9c959559f-nwmwj\" (UID: \"922726b6-be61-448e-b8ab-6e30b7ccfcbf\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-nwmwj"
May 26 11:28:04.387393 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:04.387371    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wnxmg\" (UniqueName: \"kubernetes.io/projected/922726b6-be61-448e-b8ab-6e30b7ccfcbf-kube-api-access-wnxmg\") pod \"openshift-state-metrics-9c959559f-nwmwj\" (UID: \"922726b6-be61-448e-b8ab-6e30b7ccfcbf\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-nwmwj"
May 26 11:28:04.387661 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:04.387645    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l85fs\" (UniqueName: \"kubernetes.io/projected/dfe13380-0f06-46a6-b295-86f0bc1694a4-kube-api-access-l85fs\") pod \"kube-state-metrics-54fd6bc5c4-88ht6\" (UID: \"dfe13380-0f06-46a6-b295-86f0bc1694a4\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-88ht6"
May 26 11:28:04.387733 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:04.387717    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sb7d2\" (UniqueName: \"kubernetes.io/projected/5ba9e31e-1334-452d-92d6-4540727e1831-kube-api-access-sb7d2\") pod \"node-exporter-f787b\" (UID: \"5ba9e31e-1334-452d-92d6-4540727e1831\") " pod="openshift-monitoring/node-exporter-f787b"
May 26 11:28:04.502130 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:04.502050    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-9c959559f-nwmwj"
May 26 11:28:04.527995 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:04.527968    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-88ht6"
May 26 11:28:04.634980 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:04.634951    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-9c959559f-nwmwj"]
May 26 11:28:04.638445 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:28:04.638414    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod922726b6_be61_448e_b8ab_6e30b7ccfcbf.slice/crio-65cbeaec852618da0d50296a314dcb8df07b467c92133934d327400fc0db2879 WatchSource:0}: Error finding container 65cbeaec852618da0d50296a314dcb8df07b467c92133934d327400fc0db2879: Status 404 returned error can't find the container with id 65cbeaec852618da0d50296a314dcb8df07b467c92133934d327400fc0db2879
May 26 11:28:04.668053 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:04.667990    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-54fd6bc5c4-88ht6"]
May 26 11:28:04.670496 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:28:04.670467    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfe13380_0f06_46a6_b295_86f0bc1694a4.slice/crio-650428de5f80dbb82187c445602c9de5d05b8e4e4726a98e47fc5bc75c207a93 WatchSource:0}: Error finding container 650428de5f80dbb82187c445602c9de5d05b8e4e4726a98e47fc5bc75c207a93: Status 404 returned error can't find the container with id 650428de5f80dbb82187c445602c9de5d05b8e4e4726a98e47fc5bc75c207a93
May 26 11:28:04.878920 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:04.878829    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5ba9e31e-1334-452d-92d6-4540727e1831-node-exporter-tls\") pod \"node-exporter-f787b\" (UID: \"5ba9e31e-1334-452d-92d6-4540727e1831\") " pod="openshift-monitoring/node-exporter-f787b"
May 26 11:28:04.881177 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:04.881143    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5ba9e31e-1334-452d-92d6-4540727e1831-node-exporter-tls\") pod \"node-exporter-f787b\" (UID: \"5ba9e31e-1334-452d-92d6-4540727e1831\") " pod="openshift-monitoring/node-exporter-f787b"
May 26 11:28:05.145918 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:05.145811    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-f787b"
May 26 11:28:05.156491 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:28:05.156457    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ba9e31e_1334_452d_92d6_4540727e1831.slice/crio-308a683db52d426958d29d73a78d0add2d672eb6727ca5bd917e118442e9ceae WatchSource:0}: Error finding container 308a683db52d426958d29d73a78d0add2d672eb6727ca5bd917e118442e9ceae: Status 404 returned error can't find the container with id 308a683db52d426958d29d73a78d0add2d672eb6727ca5bd917e118442e9ceae
May 26 11:28:05.272713 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:05.272637    2572 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
May 26 11:28:05.276797 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:05.276777    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
May 26 11:28:05.280690 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:05.280659    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
May 26 11:28:05.283222 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:05.282532    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
May 26 11:28:05.283222 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:05.282847    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
May 26 11:28:05.283222 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:05.283072    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
May 26 11:28:05.283425 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:05.283303    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
May 26 11:28:05.288673 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:05.283661    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
May 26 11:28:05.288673 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:05.283710    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-qqvpj\""
May 26 11:28:05.288673 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:05.283898    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
May 26 11:28:05.288673 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:05.284095    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
May 26 11:28:05.288673 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:05.288483    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
May 26 11:28:05.295553 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:05.295525    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
May 26 11:28:05.355341 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:05.355305    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-f787b" event={"ID":"5ba9e31e-1334-452d-92d6-4540727e1831","Type":"ContainerStarted","Data":"308a683db52d426958d29d73a78d0add2d672eb6727ca5bd917e118442e9ceae"}
May 26 11:28:05.359143 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:05.358456    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-9c959559f-nwmwj" event={"ID":"922726b6-be61-448e-b8ab-6e30b7ccfcbf","Type":"ContainerStarted","Data":"6eb304b34d6e1e0aa0a141dadbfc026b14be7a06bf8ff7de3d148dc282a795b9"}
May 26 11:28:05.359143 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:05.358488    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-9c959559f-nwmwj" event={"ID":"922726b6-be61-448e-b8ab-6e30b7ccfcbf","Type":"ContainerStarted","Data":"b0fec70ce8d10a4534a951575198928418f5d101877dc3c61f1d12b3ab0ff5d4"}
May 26 11:28:05.359143 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:05.358501    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-9c959559f-nwmwj" event={"ID":"922726b6-be61-448e-b8ab-6e30b7ccfcbf","Type":"ContainerStarted","Data":"65cbeaec852618da0d50296a314dcb8df07b467c92133934d327400fc0db2879"}
May 26 11:28:05.359782 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:05.359757    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-88ht6" event={"ID":"dfe13380-0f06-46a6-b295-86f0bc1694a4","Type":"ContainerStarted","Data":"650428de5f80dbb82187c445602c9de5d05b8e4e4726a98e47fc5bc75c207a93"}
May 26 11:28:05.384517 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:05.384477    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/75506346-244a-4a4c-9d67-26fc614aa68c-config-volume\") pod \"alertmanager-main-0\" (UID: \"75506346-244a-4a4c-9d67-26fc614aa68c\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 11:28:05.384666 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:05.384524    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/75506346-244a-4a4c-9d67-26fc614aa68c-config-out\") pod \"alertmanager-main-0\" (UID: \"75506346-244a-4a4c-9d67-26fc614aa68c\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 11:28:05.384666 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:05.384549    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/75506346-244a-4a4c-9d67-26fc614aa68c-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"75506346-244a-4a4c-9d67-26fc614aa68c\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 11:28:05.384666 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:05.384583    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/75506346-244a-4a4c-9d67-26fc614aa68c-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"75506346-244a-4a4c-9d67-26fc614aa68c\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 11:28:05.384666 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:05.384621    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/75506346-244a-4a4c-9d67-26fc614aa68c-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"75506346-244a-4a4c-9d67-26fc614aa68c\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 11:28:05.384666 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:05.384651    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/75506346-244a-4a4c-9d67-26fc614aa68c-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"75506346-244a-4a4c-9d67-26fc614aa68c\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 11:28:05.384976 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:05.384675    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/75506346-244a-4a4c-9d67-26fc614aa68c-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"75506346-244a-4a4c-9d67-26fc614aa68c\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 11:28:05.384976 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:05.384728    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/75506346-244a-4a4c-9d67-26fc614aa68c-tls-assets\") pod \"alertmanager-main-0\" (UID: \"75506346-244a-4a4c-9d67-26fc614aa68c\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 11:28:05.384976 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:05.384755    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wsgwk\" (UniqueName: \"kubernetes.io/projected/75506346-244a-4a4c-9d67-26fc614aa68c-kube-api-access-wsgwk\") pod \"alertmanager-main-0\" (UID: \"75506346-244a-4a4c-9d67-26fc614aa68c\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 11:28:05.384976 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:05.384791    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/75506346-244a-4a4c-9d67-26fc614aa68c-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"75506346-244a-4a4c-9d67-26fc614aa68c\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 11:28:05.384976 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:05.384857    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/75506346-244a-4a4c-9d67-26fc614aa68c-web-config\") pod \"alertmanager-main-0\" (UID: \"75506346-244a-4a4c-9d67-26fc614aa68c\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 11:28:05.385272 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:05.384996    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/75506346-244a-4a4c-9d67-26fc614aa68c-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"75506346-244a-4a4c-9d67-26fc614aa68c\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 11:28:05.385272 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:05.385034    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/75506346-244a-4a4c-9d67-26fc614aa68c-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"75506346-244a-4a4c-9d67-26fc614aa68c\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 11:28:05.486275 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:05.486239    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/75506346-244a-4a4c-9d67-26fc614aa68c-config-volume\") pod \"alertmanager-main-0\" (UID: \"75506346-244a-4a4c-9d67-26fc614aa68c\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 11:28:05.486275 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:05.486279    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/75506346-244a-4a4c-9d67-26fc614aa68c-config-out\") pod \"alertmanager-main-0\" (UID: \"75506346-244a-4a4c-9d67-26fc614aa68c\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 11:28:05.486512 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:05.486301    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/75506346-244a-4a4c-9d67-26fc614aa68c-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"75506346-244a-4a4c-9d67-26fc614aa68c\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 11:28:05.486512 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:05.486399    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/75506346-244a-4a4c-9d67-26fc614aa68c-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"75506346-244a-4a4c-9d67-26fc614aa68c\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 11:28:05.486512 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:05.486443    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/75506346-244a-4a4c-9d67-26fc614aa68c-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"75506346-244a-4a4c-9d67-26fc614aa68c\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 11:28:05.486512 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:05.486473    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/75506346-244a-4a4c-9d67-26fc614aa68c-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"75506346-244a-4a4c-9d67-26fc614aa68c\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 11:28:05.486512 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:05.486494    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/75506346-244a-4a4c-9d67-26fc614aa68c-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"75506346-244a-4a4c-9d67-26fc614aa68c\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 11:28:05.486750 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:05.486543    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/75506346-244a-4a4c-9d67-26fc614aa68c-tls-assets\") pod \"alertmanager-main-0\" (UID: \"75506346-244a-4a4c-9d67-26fc614aa68c\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 11:28:05.486750 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:05.486568    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-wsgwk\" (UniqueName: \"kubernetes.io/projected/75506346-244a-4a4c-9d67-26fc614aa68c-kube-api-access-wsgwk\") pod \"alertmanager-main-0\" (UID: \"75506346-244a-4a4c-9d67-26fc614aa68c\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 11:28:05.486750 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:05.486604    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/75506346-244a-4a4c-9d67-26fc614aa68c-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"75506346-244a-4a4c-9d67-26fc614aa68c\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 11:28:05.486750 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:05.486665    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/75506346-244a-4a4c-9d67-26fc614aa68c-web-config\") pod \"alertmanager-main-0\" (UID: \"75506346-244a-4a4c-9d67-26fc614aa68c\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 11:28:05.486750 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:05.486715    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/75506346-244a-4a4c-9d67-26fc614aa68c-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"75506346-244a-4a4c-9d67-26fc614aa68c\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 11:28:05.486750 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:05.486747    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/75506346-244a-4a4c-9d67-26fc614aa68c-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"75506346-244a-4a4c-9d67-26fc614aa68c\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 11:28:05.487520 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:05.487137    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/75506346-244a-4a4c-9d67-26fc614aa68c-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"75506346-244a-4a4c-9d67-26fc614aa68c\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 11:28:05.487651 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:05.487575    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/75506346-244a-4a4c-9d67-26fc614aa68c-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"75506346-244a-4a4c-9d67-26fc614aa68c\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 11:28:05.487716 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:28:05.487681    2572 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found
May 26 11:28:05.487772 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:28:05.487736    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/75506346-244a-4a4c-9d67-26fc614aa68c-secret-alertmanager-main-tls podName:75506346-244a-4a4c-9d67-26fc614aa68c nodeName:}" failed. No retries permitted until 2026-05-26 11:28:05.987719034 +0000 UTC m=+208.889929344 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/75506346-244a-4a4c-9d67-26fc614aa68c-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "75506346-244a-4a4c-9d67-26fc614aa68c") : secret "alertmanager-main-tls" not found
May 26 11:28:05.489576 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:05.489545    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/75506346-244a-4a4c-9d67-26fc614aa68c-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"75506346-244a-4a4c-9d67-26fc614aa68c\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 11:28:05.490431 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:05.489925    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/75506346-244a-4a4c-9d67-26fc614aa68c-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"75506346-244a-4a4c-9d67-26fc614aa68c\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 11:28:05.490431 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:05.490342    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/75506346-244a-4a4c-9d67-26fc614aa68c-config-out\") pod \"alertmanager-main-0\" (UID: \"75506346-244a-4a4c-9d67-26fc614aa68c\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 11:28:05.490431 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:05.490381    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/75506346-244a-4a4c-9d67-26fc614aa68c-web-config\") pod \"alertmanager-main-0\" (UID: \"75506346-244a-4a4c-9d67-26fc614aa68c\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 11:28:05.490431 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:05.490381    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/75506346-244a-4a4c-9d67-26fc614aa68c-config-volume\") pod \"alertmanager-main-0\" (UID: \"75506346-244a-4a4c-9d67-26fc614aa68c\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 11:28:05.490687 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:05.490477    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/75506346-244a-4a4c-9d67-26fc614aa68c-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"75506346-244a-4a4c-9d67-26fc614aa68c\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 11:28:05.491689 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:05.491663    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/75506346-244a-4a4c-9d67-26fc614aa68c-tls-assets\") pod \"alertmanager-main-0\" (UID: \"75506346-244a-4a4c-9d67-26fc614aa68c\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 11:28:05.492145 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:05.492052    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/75506346-244a-4a4c-9d67-26fc614aa68c-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"75506346-244a-4a4c-9d67-26fc614aa68c\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 11:28:05.492335 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:05.492306    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/75506346-244a-4a4c-9d67-26fc614aa68c-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"75506346-244a-4a4c-9d67-26fc614aa68c\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 11:28:05.498544 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:05.498518    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wsgwk\" (UniqueName: \"kubernetes.io/projected/75506346-244a-4a4c-9d67-26fc614aa68c-kube-api-access-wsgwk\") pod \"alertmanager-main-0\" (UID: \"75506346-244a-4a4c-9d67-26fc614aa68c\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 11:28:05.671891 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:05.671858    2572 scope.go:117] "RemoveContainer" containerID="f44c90af71b6f623ba5c71c914aa721bae260713bd3952a240ac00d781181745"
May 26 11:28:05.672216 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:28:05.672148    2572 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-76dff48d59-5bfgg_openshift-console-operator(c6373342-dd07-4497-a4f3-978ea9613b6e)\"" pod="openshift-console-operator/console-operator-76dff48d59-5bfgg" podUID="c6373342-dd07-4497-a4f3-978ea9613b6e"
May 26 11:28:05.992080 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:05.992038    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/75506346-244a-4a4c-9d67-26fc614aa68c-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"75506346-244a-4a4c-9d67-26fc614aa68c\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 11:28:05.995287 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:05.995242    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/75506346-244a-4a4c-9d67-26fc614aa68c-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"75506346-244a-4a4c-9d67-26fc614aa68c\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 11:28:06.192199 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:06.192156    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
May 26 11:28:06.364071 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:06.359717    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
May 26 11:28:06.369437 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:06.369407    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-9c959559f-nwmwj" event={"ID":"922726b6-be61-448e-b8ab-6e30b7ccfcbf","Type":"ContainerStarted","Data":"a1bb8368dd581593bd938f0503a814a723b1fedd96194de213ceef0b8aef83ba"}
May 26 11:28:06.371530 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:06.371491    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-88ht6" event={"ID":"dfe13380-0f06-46a6-b295-86f0bc1694a4","Type":"ContainerStarted","Data":"d5288362984c21b5d3998241170ed7d46350d513e97d81a21a704995edcd26bd"}
May 26 11:28:06.371530 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:06.371525    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-88ht6" event={"ID":"dfe13380-0f06-46a6-b295-86f0bc1694a4","Type":"ContainerStarted","Data":"6c13a36dd3610c723a4d4975e53e19e21674e3e42fedd7babc0b406edda19482"}
May 26 11:28:06.371689 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:06.371539    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-88ht6" event={"ID":"dfe13380-0f06-46a6-b295-86f0bc1694a4","Type":"ContainerStarted","Data":"8b16a6659fe6da5bf2946dc79b13f48e5e8f533936556dfa5ad6a497eea8b26e"}
May 26 11:28:06.394840 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:06.394795    2572 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-9c959559f-nwmwj" podStartSLOduration=1.195857691 podStartE2EDuration="2.394776636s" podCreationTimestamp="2026-05-26 11:28:04 +0000 UTC" firstStartedPulling="2026-05-26 11:28:04.750664029 +0000 UTC m=+207.652874335" lastFinishedPulling="2026-05-26 11:28:05.949582973 +0000 UTC m=+208.851793280" observedRunningTime="2026-05-26 11:28:06.393288933 +0000 UTC m=+209.295499264" watchObservedRunningTime="2026-05-26 11:28:06.394776636 +0000 UTC m=+209.296986967"
May 26 11:28:06.414755 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:06.414701    2572 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-88ht6" podStartSLOduration=1.1381104930000001 podStartE2EDuration="2.414682879s" podCreationTimestamp="2026-05-26 11:28:04 +0000 UTC" firstStartedPulling="2026-05-26 11:28:04.672328671 +0000 UTC m=+207.574538986" lastFinishedPulling="2026-05-26 11:28:05.948901052 +0000 UTC m=+208.851111372" observedRunningTime="2026-05-26 11:28:06.412873834 +0000 UTC m=+209.315084165" watchObservedRunningTime="2026-05-26 11:28:06.414682879 +0000 UTC m=+209.316893209"
May 26 11:28:06.522361 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:28:06.522278    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75506346_244a_4a4c_9d67_26fc614aa68c.slice/crio-4956e9858ae04d3305e9a75dc047eb25f9a1fb287eea5ae298c927cf281eb698 WatchSource:0}: Error finding container 4956e9858ae04d3305e9a75dc047eb25f9a1fb287eea5ae298c927cf281eb698: Status 404 returned error can't find the container with id 4956e9858ae04d3305e9a75dc047eb25f9a1fb287eea5ae298c927cf281eb698
May 26 11:28:07.378837 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:07.378795    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"75506346-244a-4a4c-9d67-26fc614aa68c","Type":"ContainerStarted","Data":"4956e9858ae04d3305e9a75dc047eb25f9a1fb287eea5ae298c927cf281eb698"}
May 26 11:28:07.380112 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:07.380086    2572 generic.go:358] "Generic (PLEG): container finished" podID="5ba9e31e-1334-452d-92d6-4540727e1831" containerID="1fc9496145bebf00d29533dcb25a6570b9e628f3d697948c766fa1667ba5c08c" exitCode=0
May 26 11:28:07.380233 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:07.380169    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-f787b" event={"ID":"5ba9e31e-1334-452d-92d6-4540727e1831","Type":"ContainerDied","Data":"1fc9496145bebf00d29533dcb25a6570b9e628f3d697948c766fa1667ba5c08c"}
May 26 11:28:08.385536 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:08.385493    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-f787b" event={"ID":"5ba9e31e-1334-452d-92d6-4540727e1831","Type":"ContainerStarted","Data":"5974b906e87b1499c4c865ac8617e379d39d8b0fb19cae57fc76b93b71d15c62"}
May 26 11:28:08.385536 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:08.385542    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-f787b" event={"ID":"5ba9e31e-1334-452d-92d6-4540727e1831","Type":"ContainerStarted","Data":"6eea4d2610f9af516e490409df878deb9224ac0b48bc670b53bc3d61420315fa"}
May 26 11:28:08.386880 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:08.386855    2572 generic.go:358] "Generic (PLEG): container finished" podID="75506346-244a-4a4c-9d67-26fc614aa68c" containerID="fbb914492c060bdd611c8af572f955631623da33eb61587b763056fdae950b74" exitCode=0
May 26 11:28:08.386998 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:08.386921    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"75506346-244a-4a4c-9d67-26fc614aa68c","Type":"ContainerDied","Data":"fbb914492c060bdd611c8af572f955631623da33eb61587b763056fdae950b74"}
May 26 11:28:08.406144 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:08.406097    2572 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-f787b" podStartSLOduration=3.019798683 podStartE2EDuration="4.406083492s" podCreationTimestamp="2026-05-26 11:28:04 +0000 UTC" firstStartedPulling="2026-05-26 11:28:05.158538958 +0000 UTC m=+208.060749276" lastFinishedPulling="2026-05-26 11:28:06.544823759 +0000 UTC m=+209.447034085" observedRunningTime="2026-05-26 11:28:08.404685698 +0000 UTC m=+211.306896041" watchObservedRunningTime="2026-05-26 11:28:08.406083492 +0000 UTC m=+211.308293820"
May 26 11:28:09.503218 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:09.500661    2572 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-6f45bf55f6-wjpfk"]
May 26 11:28:09.507696 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:09.507668    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-6f45bf55f6-wjpfk"
May 26 11:28:09.511408 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:09.511274    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
May 26 11:28:09.511408 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:09.511295    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
May 26 11:28:09.511408 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:09.511274    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
May 26 11:28:09.511408 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:09.511331    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
May 26 11:28:09.511707 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:09.511645    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-gnjbx\""
May 26 11:28:09.512175 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:09.512154    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
May 26 11:28:09.517001 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:09.516964    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
May 26 11:28:09.519825 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:09.519799    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-6f45bf55f6-wjpfk"]
May 26 11:28:09.629989 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:09.629952    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/65ccb36e-1583-405b-845d-401b170c81cd-metrics-client-ca\") pod \"telemeter-client-6f45bf55f6-wjpfk\" (UID: \"65ccb36e-1583-405b-845d-401b170c81cd\") " pod="openshift-monitoring/telemeter-client-6f45bf55f6-wjpfk"
May 26 11:28:09.630146 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:09.629998    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/65ccb36e-1583-405b-845d-401b170c81cd-telemeter-client-tls\") pod \"telemeter-client-6f45bf55f6-wjpfk\" (UID: \"65ccb36e-1583-405b-845d-401b170c81cd\") " pod="openshift-monitoring/telemeter-client-6f45bf55f6-wjpfk"
May 26 11:28:09.630146 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:09.630094    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p97dx\" (UniqueName: \"kubernetes.io/projected/65ccb36e-1583-405b-845d-401b170c81cd-kube-api-access-p97dx\") pod \"telemeter-client-6f45bf55f6-wjpfk\" (UID: \"65ccb36e-1583-405b-845d-401b170c81cd\") " pod="openshift-monitoring/telemeter-client-6f45bf55f6-wjpfk"
May 26 11:28:09.630242 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:09.630222    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/65ccb36e-1583-405b-845d-401b170c81cd-secret-telemeter-client\") pod \"telemeter-client-6f45bf55f6-wjpfk\" (UID: \"65ccb36e-1583-405b-845d-401b170c81cd\") " pod="openshift-monitoring/telemeter-client-6f45bf55f6-wjpfk"
May 26 11:28:09.630285 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:09.630271    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/65ccb36e-1583-405b-845d-401b170c81cd-federate-client-tls\") pod \"telemeter-client-6f45bf55f6-wjpfk\" (UID: \"65ccb36e-1583-405b-845d-401b170c81cd\") " pod="openshift-monitoring/telemeter-client-6f45bf55f6-wjpfk"
May 26 11:28:09.630321 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:09.630296    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/65ccb36e-1583-405b-845d-401b170c81cd-serving-certs-ca-bundle\") pod \"telemeter-client-6f45bf55f6-wjpfk\" (UID: \"65ccb36e-1583-405b-845d-401b170c81cd\") " pod="openshift-monitoring/telemeter-client-6f45bf55f6-wjpfk"
May 26 11:28:09.630321 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:09.630315    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/65ccb36e-1583-405b-845d-401b170c81cd-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6f45bf55f6-wjpfk\" (UID: \"65ccb36e-1583-405b-845d-401b170c81cd\") " pod="openshift-monitoring/telemeter-client-6f45bf55f6-wjpfk"
May 26 11:28:09.630391 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:09.630333    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/65ccb36e-1583-405b-845d-401b170c81cd-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6f45bf55f6-wjpfk\" (UID: \"65ccb36e-1583-405b-845d-401b170c81cd\") " pod="openshift-monitoring/telemeter-client-6f45bf55f6-wjpfk"
May 26 11:28:09.731149 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:09.731123    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/65ccb36e-1583-405b-845d-401b170c81cd-secret-telemeter-client\") pod \"telemeter-client-6f45bf55f6-wjpfk\" (UID: \"65ccb36e-1583-405b-845d-401b170c81cd\") " pod="openshift-monitoring/telemeter-client-6f45bf55f6-wjpfk"
May 26 11:28:09.731271 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:09.731202    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/65ccb36e-1583-405b-845d-401b170c81cd-federate-client-tls\") pod \"telemeter-client-6f45bf55f6-wjpfk\" (UID: \"65ccb36e-1583-405b-845d-401b170c81cd\") " pod="openshift-monitoring/telemeter-client-6f45bf55f6-wjpfk"
May 26 11:28:09.731271 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:09.731239    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/65ccb36e-1583-405b-845d-401b170c81cd-serving-certs-ca-bundle\") pod \"telemeter-client-6f45bf55f6-wjpfk\" (UID: \"65ccb36e-1583-405b-845d-401b170c81cd\") " pod="openshift-monitoring/telemeter-client-6f45bf55f6-wjpfk"
May 26 11:28:09.731377 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:09.731272    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/65ccb36e-1583-405b-845d-401b170c81cd-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6f45bf55f6-wjpfk\" (UID: \"65ccb36e-1583-405b-845d-401b170c81cd\") " pod="openshift-monitoring/telemeter-client-6f45bf55f6-wjpfk"
May 26 11:28:09.731377 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:09.731303    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/65ccb36e-1583-405b-845d-401b170c81cd-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6f45bf55f6-wjpfk\" (UID: \"65ccb36e-1583-405b-845d-401b170c81cd\") " pod="openshift-monitoring/telemeter-client-6f45bf55f6-wjpfk"
May 26 11:28:09.731377 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:09.731370    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/65ccb36e-1583-405b-845d-401b170c81cd-metrics-client-ca\") pod \"telemeter-client-6f45bf55f6-wjpfk\" (UID: \"65ccb36e-1583-405b-845d-401b170c81cd\") " pod="openshift-monitoring/telemeter-client-6f45bf55f6-wjpfk"
May 26 11:28:09.731525 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:09.731399    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/65ccb36e-1583-405b-845d-401b170c81cd-telemeter-client-tls\") pod \"telemeter-client-6f45bf55f6-wjpfk\" (UID: \"65ccb36e-1583-405b-845d-401b170c81cd\") " pod="openshift-monitoring/telemeter-client-6f45bf55f6-wjpfk"
May 26 11:28:09.731525 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:09.731458    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-p97dx\" (UniqueName: \"kubernetes.io/projected/65ccb36e-1583-405b-845d-401b170c81cd-kube-api-access-p97dx\") pod \"telemeter-client-6f45bf55f6-wjpfk\" (UID: \"65ccb36e-1583-405b-845d-401b170c81cd\") " pod="openshift-monitoring/telemeter-client-6f45bf55f6-wjpfk"
May 26 11:28:09.731998 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:09.731970    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/65ccb36e-1583-405b-845d-401b170c81cd-serving-certs-ca-bundle\") pod \"telemeter-client-6f45bf55f6-wjpfk\" (UID: \"65ccb36e-1583-405b-845d-401b170c81cd\") " pod="openshift-monitoring/telemeter-client-6f45bf55f6-wjpfk"
May 26 11:28:09.732247 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:09.732223    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/65ccb36e-1583-405b-845d-401b170c81cd-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6f45bf55f6-wjpfk\" (UID: \"65ccb36e-1583-405b-845d-401b170c81cd\") " pod="openshift-monitoring/telemeter-client-6f45bf55f6-wjpfk"
May 26 11:28:09.732392 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:09.732334    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/65ccb36e-1583-405b-845d-401b170c81cd-metrics-client-ca\") pod \"telemeter-client-6f45bf55f6-wjpfk\" (UID: \"65ccb36e-1583-405b-845d-401b170c81cd\") " pod="openshift-monitoring/telemeter-client-6f45bf55f6-wjpfk"
May 26 11:28:09.733954 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:09.733929    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/65ccb36e-1583-405b-845d-401b170c81cd-secret-telemeter-client\") pod \"telemeter-client-6f45bf55f6-wjpfk\" (UID: \"65ccb36e-1583-405b-845d-401b170c81cd\") " pod="openshift-monitoring/telemeter-client-6f45bf55f6-wjpfk"
May 26 11:28:09.734327 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:09.734304    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/65ccb36e-1583-405b-845d-401b170c81cd-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6f45bf55f6-wjpfk\" (UID: \"65ccb36e-1583-405b-845d-401b170c81cd\") " pod="openshift-monitoring/telemeter-client-6f45bf55f6-wjpfk"
May 26 11:28:09.734411 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:09.734392    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/65ccb36e-1583-405b-845d-401b170c81cd-federate-client-tls\") pod \"telemeter-client-6f45bf55f6-wjpfk\" (UID: \"65ccb36e-1583-405b-845d-401b170c81cd\") " pod="openshift-monitoring/telemeter-client-6f45bf55f6-wjpfk"
May 26 11:28:09.734473 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:09.734446    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/65ccb36e-1583-405b-845d-401b170c81cd-telemeter-client-tls\") pod \"telemeter-client-6f45bf55f6-wjpfk\" (UID: \"65ccb36e-1583-405b-845d-401b170c81cd\") " pod="openshift-monitoring/telemeter-client-6f45bf55f6-wjpfk"
May 26 11:28:09.740513 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:09.740493    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p97dx\" (UniqueName: \"kubernetes.io/projected/65ccb36e-1583-405b-845d-401b170c81cd-kube-api-access-p97dx\") pod \"telemeter-client-6f45bf55f6-wjpfk\" (UID: \"65ccb36e-1583-405b-845d-401b170c81cd\") " pod="openshift-monitoring/telemeter-client-6f45bf55f6-wjpfk"
May 26 11:28:09.819001 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:09.818975    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-6f45bf55f6-wjpfk"
May 26 11:28:09.963607 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:09.963581    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-6f45bf55f6-wjpfk"]
May 26 11:28:09.966341 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:28:09.966313    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65ccb36e_1583_405b_845d_401b170c81cd.slice/crio-34fb82331611afd1509cf54f0999c323adfc101a55772e9df30069f7f691fff3 WatchSource:0}: Error finding container 34fb82331611afd1509cf54f0999c323adfc101a55772e9df30069f7f691fff3: Status 404 returned error can't find the container with id 34fb82331611afd1509cf54f0999c323adfc101a55772e9df30069f7f691fff3
May 26 11:28:10.393732 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:10.393646    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6f45bf55f6-wjpfk" event={"ID":"65ccb36e-1583-405b-845d-401b170c81cd","Type":"ContainerStarted","Data":"34fb82331611afd1509cf54f0999c323adfc101a55772e9df30069f7f691fff3"}
May 26 11:28:10.396712 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:10.396676    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"75506346-244a-4a4c-9d67-26fc614aa68c","Type":"ContainerStarted","Data":"8854ea5d21c0c708da5f9afe12a5d4615fbf26a964217fa65165b6500b2cd2a0"}
May 26 11:28:10.396712 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:10.396715    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"75506346-244a-4a4c-9d67-26fc614aa68c","Type":"ContainerStarted","Data":"23e9f10ef44b6f6f3d4b0a828a5bc1b86a9ce907203441bb983fe1f1d114fc9e"}
May 26 11:28:10.396874 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:10.396728    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"75506346-244a-4a4c-9d67-26fc614aa68c","Type":"ContainerStarted","Data":"7cded0bd901003b6745b494cabac19708daa03a0afa10ce1dd5774e6bb712c88"}
May 26 11:28:10.396874 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:10.396742    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"75506346-244a-4a4c-9d67-26fc614aa68c","Type":"ContainerStarted","Data":"4a08186ca8cce4fef8e6b354e1b219ddb4875d706602b6c9a96c46bcaf8a3048"}
May 26 11:28:10.396874 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:10.396755    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"75506346-244a-4a4c-9d67-26fc614aa68c","Type":"ContainerStarted","Data":"6c6d783bacdf70f113f72dced4b5256b4f9cf053e872a53df3baea5edd0add5c"}
May 26 11:28:10.542795 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:10.542472    2572 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5898c6cd7c-xxs9f"]
May 26 11:28:10.542795 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:28:10.542745    2572 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context canceled" pod="openshift-image-registry/image-registry-5898c6cd7c-xxs9f" podUID="be4abb11-f351-4413-8833-01c4774eb128"
May 26 11:28:11.403399 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:11.403358    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"75506346-244a-4a4c-9d67-26fc614aa68c","Type":"ContainerStarted","Data":"e997ee77d8947d807666a49bbb02965ddbe5a5b5667a375a6a384f7fbf2240ac"}
May 26 11:28:11.403584 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:11.403387    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5898c6cd7c-xxs9f"
May 26 11:28:11.408306 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:11.408282    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5898c6cd7c-xxs9f"
May 26 11:28:11.447318 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:11.447263    2572 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.2614340840000002 podStartE2EDuration="6.447247944s" podCreationTimestamp="2026-05-26 11:28:05 +0000 UTC" firstStartedPulling="2026-05-26 11:28:06.524234563 +0000 UTC m=+209.426444870" lastFinishedPulling="2026-05-26 11:28:10.710048417 +0000 UTC m=+213.612258730" observedRunningTime="2026-05-26 11:28:11.443857336 +0000 UTC m=+214.346067664" watchObservedRunningTime="2026-05-26 11:28:11.447247944 +0000 UTC m=+214.349458273"
May 26 11:28:11.451640 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:11.451616    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/be4abb11-f351-4413-8833-01c4774eb128-ca-trust-extracted\") pod \"be4abb11-f351-4413-8833-01c4774eb128\" (UID: \"be4abb11-f351-4413-8833-01c4774eb128\") "
May 26 11:28:11.451768 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:11.451650    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zlkrt\" (UniqueName: \"kubernetes.io/projected/be4abb11-f351-4413-8833-01c4774eb128-kube-api-access-zlkrt\") pod \"be4abb11-f351-4413-8833-01c4774eb128\" (UID: \"be4abb11-f351-4413-8833-01c4774eb128\") "
May 26 11:28:11.451768 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:11.451686    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/be4abb11-f351-4413-8833-01c4774eb128-trusted-ca\") pod \"be4abb11-f351-4413-8833-01c4774eb128\" (UID: \"be4abb11-f351-4413-8833-01c4774eb128\") "
May 26 11:28:11.451768 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:11.451713    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/be4abb11-f351-4413-8833-01c4774eb128-installation-pull-secrets\") pod \"be4abb11-f351-4413-8833-01c4774eb128\" (UID: \"be4abb11-f351-4413-8833-01c4774eb128\") "
May 26 11:28:11.451768 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:11.451741    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/be4abb11-f351-4413-8833-01c4774eb128-registry-certificates\") pod \"be4abb11-f351-4413-8833-01c4774eb128\" (UID: \"be4abb11-f351-4413-8833-01c4774eb128\") "
May 26 11:28:11.452025 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:11.451862    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/be4abb11-f351-4413-8833-01c4774eb128-image-registry-private-configuration\") pod \"be4abb11-f351-4413-8833-01c4774eb128\" (UID: \"be4abb11-f351-4413-8833-01c4774eb128\") "
May 26 11:28:11.452025 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:11.451920    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/be4abb11-f351-4413-8833-01c4774eb128-bound-sa-token\") pod \"be4abb11-f351-4413-8833-01c4774eb128\" (UID: \"be4abb11-f351-4413-8833-01c4774eb128\") "
May 26 11:28:11.452025 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:11.451927    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be4abb11-f351-4413-8833-01c4774eb128-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "be4abb11-f351-4413-8833-01c4774eb128" (UID: "be4abb11-f351-4413-8833-01c4774eb128"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 11:28:11.452619 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:11.452214    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be4abb11-f351-4413-8833-01c4774eb128-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "be4abb11-f351-4413-8833-01c4774eb128" (UID: "be4abb11-f351-4413-8833-01c4774eb128"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 11:28:11.452619 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:11.452250    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be4abb11-f351-4413-8833-01c4774eb128-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "be4abb11-f351-4413-8833-01c4774eb128" (UID: "be4abb11-f351-4413-8833-01c4774eb128"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 11:28:11.452619 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:11.452438    2572 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/be4abb11-f351-4413-8833-01c4774eb128-trusted-ca\") on node \"ip-10-0-136-194.ec2.internal\" DevicePath \"\""
May 26 11:28:11.452619 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:11.452460    2572 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/be4abb11-f351-4413-8833-01c4774eb128-registry-certificates\") on node \"ip-10-0-136-194.ec2.internal\" DevicePath \"\""
May 26 11:28:11.452619 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:11.452478    2572 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/be4abb11-f351-4413-8833-01c4774eb128-ca-trust-extracted\") on node \"ip-10-0-136-194.ec2.internal\" DevicePath \"\""
May 26 11:28:11.454528 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:11.454498    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be4abb11-f351-4413-8833-01c4774eb128-kube-api-access-zlkrt" (OuterVolumeSpecName: "kube-api-access-zlkrt") pod "be4abb11-f351-4413-8833-01c4774eb128" (UID: "be4abb11-f351-4413-8833-01c4774eb128"). InnerVolumeSpecName "kube-api-access-zlkrt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 11:28:11.454748 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:11.454701    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be4abb11-f351-4413-8833-01c4774eb128-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "be4abb11-f351-4413-8833-01c4774eb128" (UID: "be4abb11-f351-4413-8833-01c4774eb128"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 11:28:11.454917 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:11.454890    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be4abb11-f351-4413-8833-01c4774eb128-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "be4abb11-f351-4413-8833-01c4774eb128" (UID: "be4abb11-f351-4413-8833-01c4774eb128"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 11:28:11.454983 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:11.454913    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be4abb11-f351-4413-8833-01c4774eb128-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "be4abb11-f351-4413-8833-01c4774eb128" (UID: "be4abb11-f351-4413-8833-01c4774eb128"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 11:28:11.553158 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:11.553122    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zlkrt\" (UniqueName: \"kubernetes.io/projected/be4abb11-f351-4413-8833-01c4774eb128-kube-api-access-zlkrt\") on node \"ip-10-0-136-194.ec2.internal\" DevicePath \"\""
May 26 11:28:11.553158 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:11.553153    2572 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/be4abb11-f351-4413-8833-01c4774eb128-installation-pull-secrets\") on node \"ip-10-0-136-194.ec2.internal\" DevicePath \"\""
May 26 11:28:11.553594 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:11.553169    2572 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/be4abb11-f351-4413-8833-01c4774eb128-image-registry-private-configuration\") on node \"ip-10-0-136-194.ec2.internal\" DevicePath \"\""
May 26 11:28:11.553594 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:11.553182    2572 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/be4abb11-f351-4413-8833-01c4774eb128-bound-sa-token\") on node \"ip-10-0-136-194.ec2.internal\" DevicePath \"\""
May 26 11:28:12.407896 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:12.407856    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6f45bf55f6-wjpfk" event={"ID":"65ccb36e-1583-405b-845d-401b170c81cd","Type":"ContainerStarted","Data":"cd72b8ac8359d285f80b0a203734d5d2874fe824a8c7e1d496514523efc0c836"}
May 26 11:28:12.407896 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:12.407883    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5898c6cd7c-xxs9f"
May 26 11:28:12.407896 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:12.407898    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6f45bf55f6-wjpfk" event={"ID":"65ccb36e-1583-405b-845d-401b170c81cd","Type":"ContainerStarted","Data":"7ceb248ca141467e386eb66d994c204c19047362073956e88dc0560fd7b36c8e"}
May 26 11:28:12.407896 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:12.407908    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6f45bf55f6-wjpfk" event={"ID":"65ccb36e-1583-405b-845d-401b170c81cd","Type":"ContainerStarted","Data":"c04a9194edf2a62cc430df25e74cffd462fd8b715ebd840f3bc3cff964738993"}
May 26 11:28:12.431736 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:12.431691    2572 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-6f45bf55f6-wjpfk" podStartSLOduration=1.723672064 podStartE2EDuration="3.431679432s" podCreationTimestamp="2026-05-26 11:28:09 +0000 UTC" firstStartedPulling="2026-05-26 11:28:09.968058347 +0000 UTC m=+212.870268654" lastFinishedPulling="2026-05-26 11:28:11.676065713 +0000 UTC m=+214.578276022" observedRunningTime="2026-05-26 11:28:12.430489529 +0000 UTC m=+215.332699858" watchObservedRunningTime="2026-05-26 11:28:12.431679432 +0000 UTC m=+215.333889760"
May 26 11:28:12.471950 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:12.471916    2572 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5898c6cd7c-xxs9f"]
May 26 11:28:12.481604 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:12.481580    2572 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-5898c6cd7c-xxs9f"]
May 26 11:28:12.562564 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:12.562537    2572 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/be4abb11-f351-4413-8833-01c4774eb128-registry-tls\") on node \"ip-10-0-136-194.ec2.internal\" DevicePath \"\""
May 26 11:28:13.677695 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:13.677661    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="be4abb11-f351-4413-8833-01c4774eb128" path="/var/lib/kubelet/pods/be4abb11-f351-4413-8833-01c4774eb128/volumes"
May 26 11:28:16.671454 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:16.671417    2572 scope.go:117] "RemoveContainer" containerID="f44c90af71b6f623ba5c71c914aa721bae260713bd3952a240ac00d781181745"
May 26 11:28:17.194677 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:17.194640    2572 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-57bd96b888-p8jmn"]
May 26 11:28:17.196754 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:17.196738    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-57bd96b888-p8jmn"
May 26 11:28:17.199823 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:17.199796    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
May 26 11:28:17.199940 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:17.199836    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-w2tq8\""
May 26 11:28:17.199940 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:17.199841    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
May 26 11:28:17.209537 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:17.209513    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-57bd96b888-p8jmn"]
May 26 11:28:17.306321 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:17.306289    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6ktf\" (UniqueName: \"kubernetes.io/projected/508611d9-ad49-4cd2-9e1f-e825711dd3d2-kube-api-access-z6ktf\") pod \"downloads-57bd96b888-p8jmn\" (UID: \"508611d9-ad49-4cd2-9e1f-e825711dd3d2\") " pod="openshift-console/downloads-57bd96b888-p8jmn"
May 26 11:28:17.407111 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:17.407081    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-z6ktf\" (UniqueName: \"kubernetes.io/projected/508611d9-ad49-4cd2-9e1f-e825711dd3d2-kube-api-access-z6ktf\") pod \"downloads-57bd96b888-p8jmn\" (UID: \"508611d9-ad49-4cd2-9e1f-e825711dd3d2\") " pod="openshift-console/downloads-57bd96b888-p8jmn"
May 26 11:28:17.424170 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:17.424146    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6ktf\" (UniqueName: \"kubernetes.io/projected/508611d9-ad49-4cd2-9e1f-e825711dd3d2-kube-api-access-z6ktf\") pod \"downloads-57bd96b888-p8jmn\" (UID: \"508611d9-ad49-4cd2-9e1f-e825711dd3d2\") " pod="openshift-console/downloads-57bd96b888-p8jmn"
May 26 11:28:17.427100 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:17.427081    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-5bfgg_c6373342-dd07-4497-a4f3-978ea9613b6e/console-operator/2.log"
May 26 11:28:17.427210 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:17.427169    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-76dff48d59-5bfgg" event={"ID":"c6373342-dd07-4497-a4f3-978ea9613b6e","Type":"ContainerStarted","Data":"50314d2ce95b768079eec88d567bbbd30cd21401a3a80df92a8466e1cae162b6"}
May 26 11:28:17.427478 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:17.427458    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-76dff48d59-5bfgg"
May 26 11:28:17.432426 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:17.432408    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-76dff48d59-5bfgg"
May 26 11:28:17.462425 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:17.462346    2572 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console-operator/console-operator-76dff48d59-5bfgg" podStartSLOduration=53.695454509 podStartE2EDuration="57.462333347s" podCreationTimestamp="2026-05-26 11:27:20 +0000 UTC" firstStartedPulling="2026-05-26 11:27:21.434499239 +0000 UTC m=+164.336709546" lastFinishedPulling="2026-05-26 11:27:25.201378064 +0000 UTC m=+168.103588384" observedRunningTime="2026-05-26 11:28:17.462264419 +0000 UTC m=+220.364474750" watchObservedRunningTime="2026-05-26 11:28:17.462333347 +0000 UTC m=+220.364543712"
May 26 11:28:17.506310 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:17.506282    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-57bd96b888-p8jmn"
May 26 11:28:17.641983 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:17.641959    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-57bd96b888-p8jmn"]
May 26 11:28:17.644637 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:28:17.644608    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod508611d9_ad49_4cd2_9e1f_e825711dd3d2.slice/crio-1c8ef0a045609e10ae43dc45325e13a512746b6a1467b0eb32feffcb599daf40 WatchSource:0}: Error finding container 1c8ef0a045609e10ae43dc45325e13a512746b6a1467b0eb32feffcb599daf40: Status 404 returned error can't find the container with id 1c8ef0a045609e10ae43dc45325e13a512746b6a1467b0eb32feffcb599daf40
May 26 11:28:18.431381 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:18.431342    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-57bd96b888-p8jmn" event={"ID":"508611d9-ad49-4cd2-9e1f-e825711dd3d2","Type":"ContainerStarted","Data":"1c8ef0a045609e10ae43dc45325e13a512746b6a1467b0eb32feffcb599daf40"}
May 26 11:28:34.484084 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:34.484038    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-57bd96b888-p8jmn" event={"ID":"508611d9-ad49-4cd2-9e1f-e825711dd3d2","Type":"ContainerStarted","Data":"d7c6a0dc8cdc4adf0a7600051d229b5f24405737b134dfe5b75a91f054fafaa8"}
May 26 11:28:34.484773 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:34.484749    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-57bd96b888-p8jmn"
May 26 11:28:34.493117 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:34.493090    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-57bd96b888-p8jmn"
May 26 11:28:34.508722 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:34.508665    2572 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console/downloads-57bd96b888-p8jmn" podStartSLOduration=1.731363199 podStartE2EDuration="17.508647713s" podCreationTimestamp="2026-05-26 11:28:17 +0000 UTC" firstStartedPulling="2026-05-26 11:28:17.646647517 +0000 UTC m=+220.548857825" lastFinishedPulling="2026-05-26 11:28:33.423932033 +0000 UTC m=+236.326142339" observedRunningTime="2026-05-26 11:28:34.505918364 +0000 UTC m=+237.408128693" watchObservedRunningTime="2026-05-26 11:28:34.508647713 +0000 UTC m=+237.410858043"
May 26 11:28:36.495282 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:36.495238    2572 generic.go:358] "Generic (PLEG): container finished" podID="33cb4d10-beab-4ba3-bde6-5ea8aad0a3bc" containerID="14ff270d8eef02afad6d7489bea9d419ddf2308d0354b1320f6bc39c22059b64" exitCode=0
May 26 11:28:36.495726 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:36.495322    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-78bdfb888c-g8l8m" event={"ID":"33cb4d10-beab-4ba3-bde6-5ea8aad0a3bc","Type":"ContainerDied","Data":"14ff270d8eef02afad6d7489bea9d419ddf2308d0354b1320f6bc39c22059b64"}
May 26 11:28:36.495797 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:36.495757    2572 scope.go:117] "RemoveContainer" containerID="14ff270d8eef02afad6d7489bea9d419ddf2308d0354b1320f6bc39c22059b64"
May 26 11:28:37.500832 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:37.500789    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-78bdfb888c-g8l8m" event={"ID":"33cb4d10-beab-4ba3-bde6-5ea8aad0a3bc","Type":"ContainerStarted","Data":"5ee561533b7cbe81f1e338fcb365e298665fb2701e3b4f391cd34732fd5ca0d8"}
May 26 11:28:37.572090 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:37.572061    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-4clzr_024ce274-4936-49fa-b084-ec86c46d34ed/dns-node-resolver/0.log"
May 26 11:28:46.530024 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:46.529989    2572 generic.go:358] "Generic (PLEG): container finished" podID="d2abca20-90fb-4503-b682-526829ccf2e0" containerID="787e112f98fa0f8689107e53260eaef9a446fee18fb3f59fdb04f0775e1727c4" exitCode=0
May 26 11:28:46.530491 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:46.530036    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-ds944" event={"ID":"d2abca20-90fb-4503-b682-526829ccf2e0","Type":"ContainerDied","Data":"787e112f98fa0f8689107e53260eaef9a446fee18fb3f59fdb04f0775e1727c4"}
May 26 11:28:46.530491 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:46.530315    2572 scope.go:117] "RemoveContainer" containerID="787e112f98fa0f8689107e53260eaef9a446fee18fb3f59fdb04f0775e1727c4"
May 26 11:28:47.534328 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:47.534292    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-ds944" event={"ID":"d2abca20-90fb-4503-b682-526829ccf2e0","Type":"ContainerStarted","Data":"0f68e4ccaa9016c2725f8146b8f94d7f36e61d75bdd94397fda384779a7953ee"}
May 26 11:28:49.512511 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:49.512470    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dede2966-ea61-4174-b667-c6fb46152d91-metrics-certs\") pod \"network-metrics-daemon-k7jgt\" (UID: \"dede2966-ea61-4174-b667-c6fb46152d91\") " pod="openshift-multus/network-metrics-daemon-k7jgt"
May 26 11:28:49.514789 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:49.514759    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dede2966-ea61-4174-b667-c6fb46152d91-metrics-certs\") pod \"network-metrics-daemon-k7jgt\" (UID: \"dede2966-ea61-4174-b667-c6fb46152d91\") " pod="openshift-multus/network-metrics-daemon-k7jgt"
May 26 11:28:49.573670 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:49.573639    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-2gx5q\""
May 26 11:28:49.581655 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:49.581629    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-k7jgt"
May 26 11:28:49.747583 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:49.746741    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-k7jgt"]
May 26 11:28:49.754805 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:28:49.754771    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddede2966_ea61_4174_b667_c6fb46152d91.slice/crio-47f5d8ea68240bb3bf76fab0399d6503dc0a79905a2817751f748bb8f9474352 WatchSource:0}: Error finding container 47f5d8ea68240bb3bf76fab0399d6503dc0a79905a2817751f748bb8f9474352: Status 404 returned error can't find the container with id 47f5d8ea68240bb3bf76fab0399d6503dc0a79905a2817751f748bb8f9474352
May 26 11:28:50.548327 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:50.548288    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-k7jgt" event={"ID":"dede2966-ea61-4174-b667-c6fb46152d91","Type":"ContainerStarted","Data":"47f5d8ea68240bb3bf76fab0399d6503dc0a79905a2817751f748bb8f9474352"}
May 26 11:28:52.555521 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:52.555483    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-k7jgt" event={"ID":"dede2966-ea61-4174-b667-c6fb46152d91","Type":"ContainerStarted","Data":"5d17f1d6db690df2e5b9ac8e2326ca203298661f33fc2e0e61624877d111d350"}
May 26 11:28:52.555521 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:52.555518    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-k7jgt" event={"ID":"dede2966-ea61-4174-b667-c6fb46152d91","Type":"ContainerStarted","Data":"69638f8153ea9017ed9c001a3d9e38cfed45ceae226656ef83802e3bdd01f416"}
May 26 11:28:52.572797 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:28:52.572755    2572 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-k7jgt" podStartSLOduration=253.920388055 podStartE2EDuration="4m15.57274189s" podCreationTimestamp="2026-05-26 11:24:37 +0000 UTC" firstStartedPulling="2026-05-26 11:28:49.757318351 +0000 UTC m=+252.659528657" lastFinishedPulling="2026-05-26 11:28:51.409672173 +0000 UTC m=+254.311882492" observedRunningTime="2026-05-26 11:28:52.572115138 +0000 UTC m=+255.474325478" watchObservedRunningTime="2026-05-26 11:28:52.57274189 +0000 UTC m=+255.474952232"
May 26 11:29:01.581887 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:29:01.581847    2572 generic.go:358] "Generic (PLEG): container finished" podID="29497cac-89db-463d-81fe-5baccccba06c" containerID="d0f6b755dc2935c76741779e85f377650d229dff553f09124287266ebecd4221" exitCode=0
May 26 11:29:01.582288 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:29:01.581897    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-sfz45" event={"ID":"29497cac-89db-463d-81fe-5baccccba06c","Type":"ContainerDied","Data":"d0f6b755dc2935c76741779e85f377650d229dff553f09124287266ebecd4221"}
May 26 11:29:01.582288 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:29:01.582228    2572 scope.go:117] "RemoveContainer" containerID="d0f6b755dc2935c76741779e85f377650d229dff553f09124287266ebecd4221"
May 26 11:29:02.586809 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:29:02.586767    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-sfz45" event={"ID":"29497cac-89db-463d-81fe-5baccccba06c","Type":"ContainerStarted","Data":"98e2df45288da87be356f683e26a87235bee73bd78233a019746f62a1240020a"}
May 26 11:29:17.184056 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:29:17.184018    2572 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-wb9kk" podUID="4deafc2e-1624-4384-b9b2-8a6c369c3308"
May 26 11:29:17.628277 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:29:17.628176    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-wb9kk"
May 26 11:29:21.096049 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:29:21.096005    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4deafc2e-1624-4384-b9b2-8a6c369c3308-metrics-tls\") pod \"dns-default-wb9kk\" (UID: \"4deafc2e-1624-4384-b9b2-8a6c369c3308\") " pod="openshift-dns/dns-default-wb9kk"
May 26 11:29:21.096049 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:29:21.096058    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b64f52e3-f082-4c86-a1fa-d3aa470c591f-cert\") pod \"ingress-canary-m6p64\" (UID: \"b64f52e3-f082-4c86-a1fa-d3aa470c591f\") " pod="openshift-ingress-canary/ingress-canary-m6p64"
May 26 11:29:21.098463 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:29:21.098440    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b64f52e3-f082-4c86-a1fa-d3aa470c591f-cert\") pod \"ingress-canary-m6p64\" (UID: \"b64f52e3-f082-4c86-a1fa-d3aa470c591f\") " pod="openshift-ingress-canary/ingress-canary-m6p64"
May 26 11:29:21.098463 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:29:21.098454    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4deafc2e-1624-4384-b9b2-8a6c369c3308-metrics-tls\") pod \"dns-default-wb9kk\" (UID: \"4deafc2e-1624-4384-b9b2-8a6c369c3308\") " pod="openshift-dns/dns-default-wb9kk"
May 26 11:29:21.174821 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:29:21.174788    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-rwmqw\""
May 26 11:29:21.183208 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:29:21.183165    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-m6p64"
May 26 11:29:21.231596 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:29:21.231563    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-t6qhf\""
May 26 11:29:21.239944 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:29:21.239917    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-wb9kk"
May 26 11:29:21.345268 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:29:21.345241    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-m6p64"]
May 26 11:29:21.347901 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:29:21.347817    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb64f52e3_f082_4c86_a1fa_d3aa470c591f.slice/crio-bd02bbe681eacda9ff8acce9741c6d4d16ff8da887012b0017374244e3b89065 WatchSource:0}: Error finding container bd02bbe681eacda9ff8acce9741c6d4d16ff8da887012b0017374244e3b89065: Status 404 returned error can't find the container with id bd02bbe681eacda9ff8acce9741c6d4d16ff8da887012b0017374244e3b89065
May 26 11:29:21.395483 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:29:21.395451    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-wb9kk"]
May 26 11:29:21.398589 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:29:21.398563    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4deafc2e_1624_4384_b9b2_8a6c369c3308.slice/crio-368f74592efb05cb033cb707d2b85bb94cc6444b3194ee7e46ae5215a02cf036 WatchSource:0}: Error finding container 368f74592efb05cb033cb707d2b85bb94cc6444b3194ee7e46ae5215a02cf036: Status 404 returned error can't find the container with id 368f74592efb05cb033cb707d2b85bb94cc6444b3194ee7e46ae5215a02cf036
May 26 11:29:21.642453 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:29:21.642369    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-wb9kk" event={"ID":"4deafc2e-1624-4384-b9b2-8a6c369c3308","Type":"ContainerStarted","Data":"368f74592efb05cb033cb707d2b85bb94cc6444b3194ee7e46ae5215a02cf036"}
May 26 11:29:21.643376 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:29:21.643353    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-m6p64" event={"ID":"b64f52e3-f082-4c86-a1fa-d3aa470c591f","Type":"ContainerStarted","Data":"bd02bbe681eacda9ff8acce9741c6d4d16ff8da887012b0017374244e3b89065"}
May 26 11:29:23.650417 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:29:23.650333    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-wb9kk" event={"ID":"4deafc2e-1624-4384-b9b2-8a6c369c3308","Type":"ContainerStarted","Data":"1d23ccb19ad829c7a3579d6e68a000f278af0c3eb5c1367d310f9e37849558f5"}
May 26 11:29:23.650417 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:29:23.650375    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-wb9kk" event={"ID":"4deafc2e-1624-4384-b9b2-8a6c369c3308","Type":"ContainerStarted","Data":"bd7038b18fc8312c87d407a0eda0de76313318d20f8be03f2d6895ab1bce6b01"}
May 26 11:29:23.650845 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:29:23.650560    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-wb9kk"
May 26 11:29:23.651607 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:29:23.651583    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-m6p64" event={"ID":"b64f52e3-f082-4c86-a1fa-d3aa470c591f","Type":"ContainerStarted","Data":"19b6c954a3dc1bd0ac70e0035f53d759db4ba165498f3e9110b4d599cd2c8d9b"}
May 26 11:29:23.669408 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:29:23.669366    2572 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-dns/dns-default-wb9kk" podStartSLOduration=250.845865614 podStartE2EDuration="4m12.669354286s" podCreationTimestamp="2026-05-26 11:25:11 +0000 UTC" firstStartedPulling="2026-05-26 11:29:21.400256527 +0000 UTC m=+284.302466835" lastFinishedPulling="2026-05-26 11:29:23.223745196 +0000 UTC m=+286.125955507" observedRunningTime="2026-05-26 11:29:23.667841057 +0000 UTC m=+286.570051387" watchObservedRunningTime="2026-05-26 11:29:23.669354286 +0000 UTC m=+286.571564615"
May 26 11:29:23.682599 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:29:23.682548    2572 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-m6p64" podStartSLOduration=250.806524673 podStartE2EDuration="4m12.682534433s" podCreationTimestamp="2026-05-26 11:25:11 +0000 UTC" firstStartedPulling="2026-05-26 11:29:21.349976648 +0000 UTC m=+284.252186956" lastFinishedPulling="2026-05-26 11:29:23.225986405 +0000 UTC m=+286.128196716" observedRunningTime="2026-05-26 11:29:23.682338012 +0000 UTC m=+286.584548368" watchObservedRunningTime="2026-05-26 11:29:23.682534433 +0000 UTC m=+286.584744762"
May 26 11:29:24.750695 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:29:24.750663    2572 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
May 26 11:29:24.751163 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:29:24.751130    2572 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="75506346-244a-4a4c-9d67-26fc614aa68c" containerName="kube-rbac-proxy-web" containerID="cri-o://7cded0bd901003b6745b494cabac19708daa03a0afa10ce1dd5774e6bb712c88" gracePeriod=120
May 26 11:29:24.751163 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:29:24.751148    2572 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="75506346-244a-4a4c-9d67-26fc614aa68c" containerName="kube-rbac-proxy" containerID="cri-o://23e9f10ef44b6f6f3d4b0a828a5bc1b86a9ce907203441bb983fe1f1d114fc9e" gracePeriod=120
May 26 11:29:24.751346 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:29:24.751118    2572 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="75506346-244a-4a4c-9d67-26fc614aa68c" containerName="kube-rbac-proxy-metric" containerID="cri-o://8854ea5d21c0c708da5f9afe12a5d4615fbf26a964217fa65165b6500b2cd2a0" gracePeriod=120
May 26 11:29:24.751346 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:29:24.751220    2572 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="75506346-244a-4a4c-9d67-26fc614aa68c" containerName="prom-label-proxy" containerID="cri-o://e997ee77d8947d807666a49bbb02965ddbe5a5b5667a375a6a384f7fbf2240ac" gracePeriod=120
May 26 11:29:24.751346 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:29:24.751116    2572 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="75506346-244a-4a4c-9d67-26fc614aa68c" containerName="alertmanager" containerID="cri-o://6c6d783bacdf70f113f72dced4b5256b4f9cf053e872a53df3baea5edd0add5c" gracePeriod=120
May 26 11:29:24.751346 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:29:24.751245    2572 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="75506346-244a-4a4c-9d67-26fc614aa68c" containerName="config-reloader" containerID="cri-o://4a08186ca8cce4fef8e6b354e1b219ddb4875d706602b6c9a96c46bcaf8a3048" gracePeriod=120
May 26 11:29:25.660988 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:29:25.660951    2572 generic.go:358] "Generic (PLEG): container finished" podID="75506346-244a-4a4c-9d67-26fc614aa68c" containerID="e997ee77d8947d807666a49bbb02965ddbe5a5b5667a375a6a384f7fbf2240ac" exitCode=0
May 26 11:29:25.660988 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:29:25.660977    2572 generic.go:358] "Generic (PLEG): container finished" podID="75506346-244a-4a4c-9d67-26fc614aa68c" containerID="8854ea5d21c0c708da5f9afe12a5d4615fbf26a964217fa65165b6500b2cd2a0" exitCode=0
May 26 11:29:25.660988 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:29:25.660983    2572 generic.go:358] "Generic (PLEG): container finished" podID="75506346-244a-4a4c-9d67-26fc614aa68c" containerID="23e9f10ef44b6f6f3d4b0a828a5bc1b86a9ce907203441bb983fe1f1d114fc9e" exitCode=0
May 26 11:29:25.660988 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:29:25.660989    2572 generic.go:358] "Generic (PLEG): container finished" podID="75506346-244a-4a4c-9d67-26fc614aa68c" containerID="4a08186ca8cce4fef8e6b354e1b219ddb4875d706602b6c9a96c46bcaf8a3048" exitCode=0
May 26 11:29:25.660988 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:29:25.660995    2572 generic.go:358] "Generic (PLEG): container finished" podID="75506346-244a-4a4c-9d67-26fc614aa68c" containerID="6c6d783bacdf70f113f72dced4b5256b4f9cf053e872a53df3baea5edd0add5c" exitCode=0
May 26 11:29:25.661288 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:29:25.661012    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"75506346-244a-4a4c-9d67-26fc614aa68c","Type":"ContainerDied","Data":"e997ee77d8947d807666a49bbb02965ddbe5a5b5667a375a6a384f7fbf2240ac"}
May 26 11:29:25.661288 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:29:25.661046    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"75506346-244a-4a4c-9d67-26fc614aa68c","Type":"ContainerDied","Data":"8854ea5d21c0c708da5f9afe12a5d4615fbf26a964217fa65165b6500b2cd2a0"}
May 26 11:29:25.661288 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:29:25.661056    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"75506346-244a-4a4c-9d67-26fc614aa68c","Type":"ContainerDied","Data":"23e9f10ef44b6f6f3d4b0a828a5bc1b86a9ce907203441bb983fe1f1d114fc9e"}
May 26 11:29:25.661288 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:29:25.661065    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"75506346-244a-4a4c-9d67-26fc614aa68c","Type":"ContainerDied","Data":"4a08186ca8cce4fef8e6b354e1b219ddb4875d706602b6c9a96c46bcaf8a3048"}
May 26 11:29:25.661288 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:29:25.661074    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"75506346-244a-4a4c-9d67-26fc614aa68c","Type":"ContainerDied","Data":"6c6d783bacdf70f113f72dced4b5256b4f9cf053e872a53df3baea5edd0add5c"}
May 26 11:29:25.993995 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:29:25.993970    2572 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
May 26 11:29:26.043903 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:29:26.043867    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/75506346-244a-4a4c-9d67-26fc614aa68c-secret-alertmanager-kube-rbac-proxy\") pod \"75506346-244a-4a4c-9d67-26fc614aa68c\" (UID: \"75506346-244a-4a4c-9d67-26fc614aa68c\") "
May 26 11:29:26.044059 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:29:26.043929    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/75506346-244a-4a4c-9d67-26fc614aa68c-alertmanager-main-db\") pod \"75506346-244a-4a4c-9d67-26fc614aa68c\" (UID: \"75506346-244a-4a4c-9d67-26fc614aa68c\") "
May 26 11:29:26.044116 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:29:26.044066    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/75506346-244a-4a4c-9d67-26fc614aa68c-metrics-client-ca\") pod \"75506346-244a-4a4c-9d67-26fc614aa68c\" (UID: \"75506346-244a-4a4c-9d67-26fc614aa68c\") "
May 26 11:29:26.044116 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:29:26.044104    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/75506346-244a-4a4c-9d67-26fc614aa68c-secret-alertmanager-kube-rbac-proxy-web\") pod \"75506346-244a-4a4c-9d67-26fc614aa68c\" (UID: \"75506346-244a-4a4c-9d67-26fc614aa68c\") "
May 26 11:29:26.044296 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:29:26.044121    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/75506346-244a-4a4c-9d67-26fc614aa68c-config-volume\") pod \"75506346-244a-4a4c-9d67-26fc614aa68c\" (UID: \"75506346-244a-4a4c-9d67-26fc614aa68c\") "
May 26 11:29:26.044296 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:29:26.044156    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/75506346-244a-4a4c-9d67-26fc614aa68c-secret-alertmanager-main-tls\") pod \"75506346-244a-4a4c-9d67-26fc614aa68c\" (UID: \"75506346-244a-4a4c-9d67-26fc614aa68c\") "
May 26 11:29:26.044296 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:29:26.044269    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75506346-244a-4a4c-9d67-26fc614aa68c-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "75506346-244a-4a4c-9d67-26fc614aa68c" (UID: "75506346-244a-4a4c-9d67-26fc614aa68c"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 11:29:26.044459 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:29:26.044308    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/75506346-244a-4a4c-9d67-26fc614aa68c-tls-assets\") pod \"75506346-244a-4a4c-9d67-26fc614aa68c\" (UID: \"75506346-244a-4a4c-9d67-26fc614aa68c\") "
May 26 11:29:26.044459 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:29:26.044362    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/75506346-244a-4a4c-9d67-26fc614aa68c-cluster-tls-config\") pod \"75506346-244a-4a4c-9d67-26fc614aa68c\" (UID: \"75506346-244a-4a4c-9d67-26fc614aa68c\") "
May 26 11:29:26.044459 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:29:26.044395    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/75506346-244a-4a4c-9d67-26fc614aa68c-alertmanager-trusted-ca-bundle\") pod \"75506346-244a-4a4c-9d67-26fc614aa68c\" (UID: \"75506346-244a-4a4c-9d67-26fc614aa68c\") "
May 26 11:29:26.044459 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:29:26.044434    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wsgwk\" (UniqueName: \"kubernetes.io/projected/75506346-244a-4a4c-9d67-26fc614aa68c-kube-api-access-wsgwk\") pod \"75506346-244a-4a4c-9d67-26fc614aa68c\" (UID: \"75506346-244a-4a4c-9d67-26fc614aa68c\") "
May 26 11:29:26.044642 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:29:26.044457    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75506346-244a-4a4c-9d67-26fc614aa68c-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "75506346-244a-4a4c-9d67-26fc614aa68c" (UID: "75506346-244a-4a4c-9d67-26fc614aa68c"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 11:29:26.044642 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:29:26.044486    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/75506346-244a-4a4c-9d67-26fc614aa68c-secret-alertmanager-kube-rbac-proxy-metric\") pod \"75506346-244a-4a4c-9d67-26fc614aa68c\" (UID: \"75506346-244a-4a4c-9d67-26fc614aa68c\") "
May 26 11:29:26.044642 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:29:26.044512    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/75506346-244a-4a4c-9d67-26fc614aa68c-web-config\") pod \"75506346-244a-4a4c-9d67-26fc614aa68c\" (UID: \"75506346-244a-4a4c-9d67-26fc614aa68c\") "
May 26 11:29:26.044642 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:29:26.044554    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/75506346-244a-4a4c-9d67-26fc614aa68c-config-out\") pod \"75506346-244a-4a4c-9d67-26fc614aa68c\" (UID: \"75506346-244a-4a4c-9d67-26fc614aa68c\") "
May 26 11:29:26.044833 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:29:26.044814    2572 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/75506346-244a-4a4c-9d67-26fc614aa68c-metrics-client-ca\") on node \"ip-10-0-136-194.ec2.internal\" DevicePath \"\""
May 26 11:29:26.044888 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:29:26.044836    2572 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/75506346-244a-4a4c-9d67-26fc614aa68c-alertmanager-main-db\") on node \"ip-10-0-136-194.ec2.internal\" DevicePath \"\""
May 26 11:29:26.045368 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:29:26.045328    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75506346-244a-4a4c-9d67-26fc614aa68c-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "75506346-244a-4a4c-9d67-26fc614aa68c" (UID: "75506346-244a-4a4c-9d67-26fc614aa68c"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 11:29:26.048562 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:29:26.048510    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75506346-244a-4a4c-9d67-26fc614aa68c-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "75506346-244a-4a4c-9d67-26fc614aa68c" (UID: "75506346-244a-4a4c-9d67-26fc614aa68c"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 11:29:26.049609 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:29:26.049563    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75506346-244a-4a4c-9d67-26fc614aa68c-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "75506346-244a-4a4c-9d67-26fc614aa68c" (UID: "75506346-244a-4a4c-9d67-26fc614aa68c"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 11:29:26.050304 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:29:26.049813    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75506346-244a-4a4c-9d67-26fc614aa68c-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "75506346-244a-4a4c-9d67-26fc614aa68c" (UID: "75506346-244a-4a4c-9d67-26fc614aa68c"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 11:29:26.050304 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:29:26.049909    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75506346-244a-4a4c-9d67-26fc614aa68c-kube-api-access-wsgwk" (OuterVolumeSpecName: "kube-api-access-wsgwk") pod "75506346-244a-4a4c-9d67-26fc614aa68c" (UID: "75506346-244a-4a4c-9d67-26fc614aa68c"). InnerVolumeSpecName "kube-api-access-wsgwk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 11:29:26.050304 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:29:26.050073    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75506346-244a-4a4c-9d67-26fc614aa68c-config-out" (OuterVolumeSpecName: "config-out") pod "75506346-244a-4a4c-9d67-26fc614aa68c" (UID: "75506346-244a-4a4c-9d67-26fc614aa68c"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 11:29:26.050304 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:29:26.050211    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75506346-244a-4a4c-9d67-26fc614aa68c-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "75506346-244a-4a4c-9d67-26fc614aa68c" (UID: "75506346-244a-4a4c-9d67-26fc614aa68c"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 11:29:26.050993 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:29:26.050965    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75506346-244a-4a4c-9d67-26fc614aa68c-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "75506346-244a-4a4c-9d67-26fc614aa68c" (UID: "75506346-244a-4a4c-9d67-26fc614aa68c"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 11:29:26.051680 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:29:26.051654    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75506346-244a-4a4c-9d67-26fc614aa68c-config-volume" (OuterVolumeSpecName: "config-volume") pod "75506346-244a-4a4c-9d67-26fc614aa68c" (UID: "75506346-244a-4a4c-9d67-26fc614aa68c"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 11:29:26.054571 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:29:26.054536    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75506346-244a-4a4c-9d67-26fc614aa68c-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "75506346-244a-4a4c-9d67-26fc614aa68c" (UID: "75506346-244a-4a4c-9d67-26fc614aa68c"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 11:29:26.062934 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:29:26.062900    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75506346-244a-4a4c-9d67-26fc614aa68c-web-config" (OuterVolumeSpecName: "web-config") pod "75506346-244a-4a4c-9d67-26fc614aa68c" (UID: "75506346-244a-4a4c-9d67-26fc614aa68c"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 11:29:26.145459 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:29:26.145428    2572 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/75506346-244a-4a4c-9d67-26fc614aa68c-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-136-194.ec2.internal\" DevicePath \"\""
May 26 11:29:26.145459 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:29:26.145455    2572 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/75506346-244a-4a4c-9d67-26fc614aa68c-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-136-194.ec2.internal\" DevicePath \"\""
May 26 11:29:26.145644 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:29:26.145468    2572 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/75506346-244a-4a4c-9d67-26fc614aa68c-config-volume\") on node \"ip-10-0-136-194.ec2.internal\" DevicePath \"\""
May 26 11:29:26.145644 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:29:26.145477    2572 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/75506346-244a-4a4c-9d67-26fc614aa68c-secret-alertmanager-main-tls\") on node \"ip-10-0-136-194.ec2.internal\" DevicePath \"\""
May 26 11:29:26.145644 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:29:26.145487    2572 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/75506346-244a-4a4c-9d67-26fc614aa68c-tls-assets\") on node \"ip-10-0-136-194.ec2.internal\" DevicePath \"\""
May 26 11:29:26.145644 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:29:26.145497    2572 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/75506346-244a-4a4c-9d67-26fc614aa68c-cluster-tls-config\") on node \"ip-10-0-136-194.ec2.internal\" DevicePath \"\""
May 26 11:29:26.145644 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:29:26.145506    2572 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/75506346-244a-4a4c-9d67-26fc614aa68c-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-136-194.ec2.internal\" DevicePath \"\""
May 26 11:29:26.145644 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:29:26.145514    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wsgwk\" (UniqueName: \"kubernetes.io/projected/75506346-244a-4a4c-9d67-26fc614aa68c-kube-api-access-wsgwk\") on node \"ip-10-0-136-194.ec2.internal\" DevicePath \"\""
May 26 11:29:26.145644 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:29:26.145523    2572 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/75506346-244a-4a4c-9d67-26fc614aa68c-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-136-194.ec2.internal\" DevicePath \"\""
May 26 11:29:26.145644 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:29:26.145534    2572 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/75506346-244a-4a4c-9d67-26fc614aa68c-web-config\") on node \"ip-10-0-136-194.ec2.internal\" DevicePath \"\""
May 26 11:29:26.145644 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:29:26.145542    2572 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/75506346-244a-4a4c-9d67-26fc614aa68c-config-out\") on node \"ip-10-0-136-194.ec2.internal\" DevicePath \"\""
May 26 11:29:26.667290 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:29:26.667258    2572 generic.go:358] "Generic (PLEG): container finished" podID="75506346-244a-4a4c-9d67-26fc614aa68c" containerID="7cded0bd901003b6745b494cabac19708daa03a0afa10ce1dd5774e6bb712c88" exitCode=0
May 26 11:29:26.667290 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:29:26.667295    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"75506346-244a-4a4c-9d67-26fc614aa68c","Type":"ContainerDied","Data":"7cded0bd901003b6745b494cabac19708daa03a0afa10ce1dd5774e6bb712c88"}
May 26 11:29:26.667598 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:29:26.667322    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"75506346-244a-4a4c-9d67-26fc614aa68c","Type":"ContainerDied","Data":"4956e9858ae04d3305e9a75dc047eb25f9a1fb287eea5ae298c927cf281eb698"}
May 26 11:29:26.667598 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:29:26.667338    2572 scope.go:117] "RemoveContainer" containerID="e997ee77d8947d807666a49bbb02965ddbe5a5b5667a375a6a384f7fbf2240ac"
May 26 11:29:26.667598 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:29:26.667389    2572 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
May 26 11:29:26.676305 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:29:26.676012    2572 scope.go:117] "RemoveContainer" containerID="8854ea5d21c0c708da5f9afe12a5d4615fbf26a964217fa65165b6500b2cd2a0"
May 26 11:29:26.682890 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:29:26.682869    2572 scope.go:117] "RemoveContainer" containerID="23e9f10ef44b6f6f3d4b0a828a5bc1b86a9ce907203441bb983fe1f1d114fc9e"
May 26 11:29:26.689491 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:29:26.689473    2572 scope.go:117] "RemoveContainer" containerID="7cded0bd901003b6745b494cabac19708daa03a0afa10ce1dd5774e6bb712c88"
May 26 11:29:26.692183 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:29:26.692154    2572 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
May 26 11:29:26.696504 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:29:26.696472    2572 scope.go:117] "RemoveContainer" containerID="4a08186ca8cce4fef8e6b354e1b219ddb4875d706602b6c9a96c46bcaf8a3048"
May 26 11:29:26.698051 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:29:26.698033    2572 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
May 26 11:29:26.703368 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:29:26.703353    2572 scope.go:117] "RemoveContainer" containerID="6c6d783bacdf70f113f72dced4b5256b4f9cf053e872a53df3baea5edd0add5c"
May 26 11:29:26.709673 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:29:26.709653    2572 scope.go:117] "RemoveContainer" containerID="fbb914492c060bdd611c8af572f955631623da33eb61587b763056fdae950b74"
May 26 11:29:26.715789 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:29:26.715774    2572 scope.go:117] "RemoveContainer" containerID="e997ee77d8947d807666a49bbb02965ddbe5a5b5667a375a6a384f7fbf2240ac"
May 26 11:29:26.716045 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:29:26.716024    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e997ee77d8947d807666a49bbb02965ddbe5a5b5667a375a6a384f7fbf2240ac\": container with ID starting with e997ee77d8947d807666a49bbb02965ddbe5a5b5667a375a6a384f7fbf2240ac not found: ID does not exist" containerID="e997ee77d8947d807666a49bbb02965ddbe5a5b5667a375a6a384f7fbf2240ac"
May 26 11:29:26.716090 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:29:26.716053    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e997ee77d8947d807666a49bbb02965ddbe5a5b5667a375a6a384f7fbf2240ac"} err="failed to get container status \"e997ee77d8947d807666a49bbb02965ddbe5a5b5667a375a6a384f7fbf2240ac\": rpc error: code = NotFound desc = could not find container \"e997ee77d8947d807666a49bbb02965ddbe5a5b5667a375a6a384f7fbf2240ac\": container with ID starting with e997ee77d8947d807666a49bbb02965ddbe5a5b5667a375a6a384f7fbf2240ac not found: ID does not exist"
May 26 11:29:26.716090 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:29:26.716083    2572 scope.go:117] "RemoveContainer" containerID="8854ea5d21c0c708da5f9afe12a5d4615fbf26a964217fa65165b6500b2cd2a0"
May 26 11:29:26.716361 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:29:26.716343    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8854ea5d21c0c708da5f9afe12a5d4615fbf26a964217fa65165b6500b2cd2a0\": container with ID starting with 8854ea5d21c0c708da5f9afe12a5d4615fbf26a964217fa65165b6500b2cd2a0 not found: ID does not exist" containerID="8854ea5d21c0c708da5f9afe12a5d4615fbf26a964217fa65165b6500b2cd2a0"
May 26 11:29:26.716420 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:29:26.716366    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8854ea5d21c0c708da5f9afe12a5d4615fbf26a964217fa65165b6500b2cd2a0"} err="failed to get container status \"8854ea5d21c0c708da5f9afe12a5d4615fbf26a964217fa65165b6500b2cd2a0\": rpc error: code = NotFound desc = could not find container \"8854ea5d21c0c708da5f9afe12a5d4615fbf26a964217fa65165b6500b2cd2a0\": container with ID starting with 8854ea5d21c0c708da5f9afe12a5d4615fbf26a964217fa65165b6500b2cd2a0 not found: ID does not exist"
May 26 11:29:26.716420 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:29:26.716382    2572 scope.go:117] "RemoveContainer" containerID="23e9f10ef44b6f6f3d4b0a828a5bc1b86a9ce907203441bb983fe1f1d114fc9e"
May 26 11:29:26.716599 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:29:26.716583    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23e9f10ef44b6f6f3d4b0a828a5bc1b86a9ce907203441bb983fe1f1d114fc9e\": container with ID starting with 23e9f10ef44b6f6f3d4b0a828a5bc1b86a9ce907203441bb983fe1f1d114fc9e not found: ID does not exist" containerID="23e9f10ef44b6f6f3d4b0a828a5bc1b86a9ce907203441bb983fe1f1d114fc9e"
May 26 11:29:26.716682 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:29:26.716607    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23e9f10ef44b6f6f3d4b0a828a5bc1b86a9ce907203441bb983fe1f1d114fc9e"} err="failed to get container status \"23e9f10ef44b6f6f3d4b0a828a5bc1b86a9ce907203441bb983fe1f1d114fc9e\": rpc error: code = NotFound desc = could not find container \"23e9f10ef44b6f6f3d4b0a828a5bc1b86a9ce907203441bb983fe1f1d114fc9e\": container with ID starting with 23e9f10ef44b6f6f3d4b0a828a5bc1b86a9ce907203441bb983fe1f1d114fc9e not found: ID does not exist"
May 26 11:29:26.716682 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:29:26.716625    2572 scope.go:117] "RemoveContainer" containerID="7cded0bd901003b6745b494cabac19708daa03a0afa10ce1dd5774e6bb712c88"
May 26 11:29:26.716862 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:29:26.716838    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7cded0bd901003b6745b494cabac19708daa03a0afa10ce1dd5774e6bb712c88\": container with ID starting with 7cded0bd901003b6745b494cabac19708daa03a0afa10ce1dd5774e6bb712c88 not found: ID does not exist" containerID="7cded0bd901003b6745b494cabac19708daa03a0afa10ce1dd5774e6bb712c88"
May 26 11:29:26.716914 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:29:26.716861    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7cded0bd901003b6745b494cabac19708daa03a0afa10ce1dd5774e6bb712c88"} err="failed to get container status \"7cded0bd901003b6745b494cabac19708daa03a0afa10ce1dd5774e6bb712c88\": rpc error: code = NotFound desc = could not find container \"7cded0bd901003b6745b494cabac19708daa03a0afa10ce1dd5774e6bb712c88\": container with ID starting with 7cded0bd901003b6745b494cabac19708daa03a0afa10ce1dd5774e6bb712c88 not found: ID does not exist"
May 26 11:29:26.716914 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:29:26.716876    2572 scope.go:117] "RemoveContainer" containerID="4a08186ca8cce4fef8e6b354e1b219ddb4875d706602b6c9a96c46bcaf8a3048"
May 26 11:29:26.717140 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:29:26.717116    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a08186ca8cce4fef8e6b354e1b219ddb4875d706602b6c9a96c46bcaf8a3048\": container with ID starting with 4a08186ca8cce4fef8e6b354e1b219ddb4875d706602b6c9a96c46bcaf8a3048 not found: ID does not exist" containerID="4a08186ca8cce4fef8e6b354e1b219ddb4875d706602b6c9a96c46bcaf8a3048"
May 26 11:29:26.717226 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:29:26.717144    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a08186ca8cce4fef8e6b354e1b219ddb4875d706602b6c9a96c46bcaf8a3048"} err="failed to get container status \"4a08186ca8cce4fef8e6b354e1b219ddb4875d706602b6c9a96c46bcaf8a3048\": rpc error: code = NotFound desc = could not find container \"4a08186ca8cce4fef8e6b354e1b219ddb4875d706602b6c9a96c46bcaf8a3048\": container with ID starting with 4a08186ca8cce4fef8e6b354e1b219ddb4875d706602b6c9a96c46bcaf8a3048 not found: ID does not exist"
May 26 11:29:26.717226 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:29:26.717164    2572 scope.go:117] "RemoveContainer" containerID="6c6d783bacdf70f113f72dced4b5256b4f9cf053e872a53df3baea5edd0add5c"
May 26 11:29:26.717476 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:29:26.717455    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c6d783bacdf70f113f72dced4b5256b4f9cf053e872a53df3baea5edd0add5c\": container with ID starting with 6c6d783bacdf70f113f72dced4b5256b4f9cf053e872a53df3baea5edd0add5c not found: ID does not exist" containerID="6c6d783bacdf70f113f72dced4b5256b4f9cf053e872a53df3baea5edd0add5c"
May 26 11:29:26.717570 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:29:26.717479    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c6d783bacdf70f113f72dced4b5256b4f9cf053e872a53df3baea5edd0add5c"} err="failed to get container status \"6c6d783bacdf70f113f72dced4b5256b4f9cf053e872a53df3baea5edd0add5c\": rpc error: code = NotFound desc = could not find container \"6c6d783bacdf70f113f72dced4b5256b4f9cf053e872a53df3baea5edd0add5c\": container with ID starting with 6c6d783bacdf70f113f72dced4b5256b4f9cf053e872a53df3baea5edd0add5c not found: ID does not exist"
May 26 11:29:26.717570 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:29:26.717499    2572 scope.go:117] "RemoveContainer" containerID="fbb914492c060bdd611c8af572f955631623da33eb61587b763056fdae950b74"
May 26 11:29:26.717759 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:29:26.717743    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fbb914492c060bdd611c8af572f955631623da33eb61587b763056fdae950b74\": container with ID starting with fbb914492c060bdd611c8af572f955631623da33eb61587b763056fdae950b74 not found: ID does not exist" containerID="fbb914492c060bdd611c8af572f955631623da33eb61587b763056fdae950b74"
May 26 11:29:26.717801 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:29:26.717765    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fbb914492c060bdd611c8af572f955631623da33eb61587b763056fdae950b74"} err="failed to get container status \"fbb914492c060bdd611c8af572f955631623da33eb61587b763056fdae950b74\": rpc error: code = NotFound desc = could not find container \"fbb914492c060bdd611c8af572f955631623da33eb61587b763056fdae950b74\": container with ID starting with fbb914492c060bdd611c8af572f955631623da33eb61587b763056fdae950b74 not found: ID does not exist"
May 26 11:29:26.732793 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:29:26.732774    2572 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
May 26 11:29:26.733091 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:29:26.733079    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75506346-244a-4a4c-9d67-26fc614aa68c" containerName="kube-rbac-proxy-metric"
May 26 11:29:26.733135 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:29:26.733093    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="75506346-244a-4a4c-9d67-26fc614aa68c" containerName="kube-rbac-proxy-metric"
May 26 11:29:26.733135 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:29:26.733101    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75506346-244a-4a4c-9d67-26fc614aa68c" containerName="prom-label-proxy"
May 26 11:29:26.733135 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:29:26.733107    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="75506346-244a-4a4c-9d67-26fc614aa68c" containerName="prom-label-proxy"
May 26 11:29:26.733135 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:29:26.733117    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75506346-244a-4a4c-9d67-26fc614aa68c" containerName="kube-rbac-proxy-web"
May 26 11:29:26.733135 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:29:26.733123    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="75506346-244a-4a4c-9d67-26fc614aa68c" containerName="kube-rbac-proxy-web"
May 26 11:29:26.733135 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:29:26.733130    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75506346-244a-4a4c-9d67-26fc614aa68c" containerName="kube-rbac-proxy"
May 26 11:29:26.733135 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:29:26.733135    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="75506346-244a-4a4c-9d67-26fc614aa68c" containerName="kube-rbac-proxy"
May 26 11:29:26.733351 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:29:26.733149    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75506346-244a-4a4c-9d67-26fc614aa68c" containerName="init-config-reloader"
May 26 11:29:26.733351 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:29:26.733154    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="75506346-244a-4a4c-9d67-26fc614aa68c" containerName="init-config-reloader"
May 26 11:29:26.733351 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:29:26.733163    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75506346-244a-4a4c-9d67-26fc614aa68c" containerName="alertmanager"
May 26 11:29:26.733351 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:29:26.733169    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="75506346-244a-4a4c-9d67-26fc614aa68c" containerName="alertmanager"
May 26 11:29:26.733351 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:29:26.733174    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75506346-244a-4a4c-9d67-26fc614aa68c" containerName="config-reloader"
May 26 11:29:26.733351 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:29:26.733179    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="75506346-244a-4a4c-9d67-26fc614aa68c" containerName="config-reloader"
May 26 11:29:26.733351 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:29:26.733240    2572 memory_manager.go:356] "RemoveStaleState removing state" podUID="75506346-244a-4a4c-9d67-26fc614aa68c" containerName="prom-label-proxy"
May 26 11:29:26.733351 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:29:26.733250    2572 memory_manager.go:356] "RemoveStaleState removing state" podUID="75506346-244a-4a4c-9d67-26fc614aa68c" containerName="kube-rbac-proxy-metric"
May 26 11:29:26.733351 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:29:26.733258    2572 memory_manager.go:356] "RemoveStaleState removing state" podUID="75506346-244a-4a4c-9d67-26fc614aa68c" containerName="alertmanager"
May 26 11:29:26.733351 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:29:26.733264    2572 memory_manager.go:356] "RemoveStaleState removing state" podUID="75506346-244a-4a4c-9d67-26fc614aa68c" containerName="config-reloader"
May 26 11:29:26.733351 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:29:26.733273    2572 memory_manager.go:356] "RemoveStaleState removing state" podUID="75506346-244a-4a4c-9d67-26fc614aa68c" containerName="kube-rbac-proxy-web"
May 26 11:29:26.733351 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:29:26.733279    2572 memory_manager.go:356] "RemoveStaleState removing state" podUID="75506346-244a-4a4c-9d67-26fc614aa68c" containerName="kube-rbac-proxy"
May 26 11:29:26.737991 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:29:26.737975    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
May 26 11:29:26.740985 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:29:26.740967    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
May 26 11:29:26.741083 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:29:26.740988    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
May 26 11:29:26.741083 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:29:26.740967    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
May 26 11:29:26.741083 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:29:26.741060    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
May 26 11:29:26.741647 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:29:26.741633    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
May 26 11:29:26.741756 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:29:26.741739    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
May 26 11:29:26.741867 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:29:26.741850    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-qqvpj\""
May 26 11:29:26.741945 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:29:26.741930    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
May 26 11:29:26.742160 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:29:26.742144    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
May 26 11:29:26.748459 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:29:26.748443    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
May 26 11:29:26.752263 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:29:26.752241    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
May 26 11:29:26.851359 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:29:26.851328    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5162c126-bea7-4999-8f87-b4f03ae8502a-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"5162c126-bea7-4999-8f87-b4f03ae8502a\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 11:29:26.851475 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:29:26.851360    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5162c126-bea7-4999-8f87-b4f03ae8502a-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"5162c126-bea7-4999-8f87-b4f03ae8502a\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 11:29:26.851475 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:29:26.851380    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5162c126-bea7-4999-8f87-b4f03ae8502a-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"5162c126-bea7-4999-8f87-b4f03ae8502a\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 11:29:26.851475 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:29:26.851442    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5162c126-bea7-4999-8f87-b4f03ae8502a-config-out\") pod \"alertmanager-main-0\" (UID: \"5162c126-bea7-4999-8f87-b4f03ae8502a\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 11:29:26.851617 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:29:26.851492    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5162c126-bea7-4999-8f87-b4f03ae8502a-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"5162c126-bea7-4999-8f87-b4f03ae8502a\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 11:29:26.851617 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:29:26.851528    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5162c126-bea7-4999-8f87-b4f03ae8502a-tls-assets\") pod \"alertmanager-main-0\" (UID: \"5162c126-bea7-4999-8f87-b4f03ae8502a\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 11:29:26.851617 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:29:26.851593    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5162c126-bea7-4999-8f87-b4f03ae8502a-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"5162c126-bea7-4999-8f87-b4f03ae8502a\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 11:29:26.851617 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:29:26.851611    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5162c126-bea7-4999-8f87-b4f03ae8502a-config-volume\") pod \"alertmanager-main-0\" (UID: \"5162c126-bea7-4999-8f87-b4f03ae8502a\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 11:29:26.851764 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:29:26.851634    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5162c126-bea7-4999-8f87-b4f03ae8502a-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"5162c126-bea7-4999-8f87-b4f03ae8502a\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 11:29:26.851764 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:29:26.851649    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5162c126-bea7-4999-8f87-b4f03ae8502a-web-config\") pod \"alertmanager-main-0\" (UID: \"5162c126-bea7-4999-8f87-b4f03ae8502a\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 11:29:26.851764 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:29:26.851665    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5162c126-bea7-4999-8f87-b4f03ae8502a-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"5162c126-bea7-4999-8f87-b4f03ae8502a\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 11:29:26.851764 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:29:26.851684    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5162c126-bea7-4999-8f87-b4f03ae8502a-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"5162c126-bea7-4999-8f87-b4f03ae8502a\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 11:29:26.851764 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:29:26.851709    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zphvc\" (UniqueName: \"kubernetes.io/projected/5162c126-bea7-4999-8f87-b4f03ae8502a-kube-api-access-zphvc\") pod \"alertmanager-main-0\" (UID: \"5162c126-bea7-4999-8f87-b4f03ae8502a\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 11:29:26.952113 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:29:26.952034    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-zphvc\" (UniqueName: \"kubernetes.io/projected/5162c126-bea7-4999-8f87-b4f03ae8502a-kube-api-access-zphvc\") pod \"alertmanager-main-0\" (UID: \"5162c126-bea7-4999-8f87-b4f03ae8502a\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 11:29:26.952113 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:29:26.952077    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5162c126-bea7-4999-8f87-b4f03ae8502a-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"5162c126-bea7-4999-8f87-b4f03ae8502a\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 11:29:26.952113 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:29:26.952098    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5162c126-bea7-4999-8f87-b4f03ae8502a-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"5162c126-bea7-4999-8f87-b4f03ae8502a\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 11:29:26.952431 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:29:26.952123    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5162c126-bea7-4999-8f87-b4f03ae8502a-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"5162c126-bea7-4999-8f87-b4f03ae8502a\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 11:29:26.952431 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:29:26.952161    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5162c126-bea7-4999-8f87-b4f03ae8502a-config-out\") pod \"alertmanager-main-0\" (UID: \"5162c126-bea7-4999-8f87-b4f03ae8502a\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 11:29:26.952431 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:29:26.952216    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5162c126-bea7-4999-8f87-b4f03ae8502a-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"5162c126-bea7-4999-8f87-b4f03ae8502a\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 11:29:26.952431 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:29:26.952246    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5162c126-bea7-4999-8f87-b4f03ae8502a-tls-assets\") pod \"alertmanager-main-0\" (UID: \"5162c126-bea7-4999-8f87-b4f03ae8502a\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 11:29:26.952431 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:29:26.952299    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5162c126-bea7-4999-8f87-b4f03ae8502a-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"5162c126-bea7-4999-8f87-b4f03ae8502a\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 11:29:26.952431 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:29:26.952417    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5162c126-bea7-4999-8f87-b4f03ae8502a-config-volume\") pod \"alertmanager-main-0\" (UID: \"5162c126-bea7-4999-8f87-b4f03ae8502a\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 11:29:26.952740 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:29:26.952479    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5162c126-bea7-4999-8f87-b4f03ae8502a-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"5162c126-bea7-4999-8f87-b4f03ae8502a\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 11:29:26.952740 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:29:26.952508    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5162c126-bea7-4999-8f87-b4f03ae8502a-web-config\") pod \"alertmanager-main-0\" (UID: \"5162c126-bea7-4999-8f87-b4f03ae8502a\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 11:29:26.952740 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:29:26.952544    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5162c126-bea7-4999-8f87-b4f03ae8502a-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"5162c126-bea7-4999-8f87-b4f03ae8502a\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 11:29:26.952740 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:29:26.952580    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5162c126-bea7-4999-8f87-b4f03ae8502a-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"5162c126-bea7-4999-8f87-b4f03ae8502a\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 11:29:26.952964 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:29:26.952935    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5162c126-bea7-4999-8f87-b4f03ae8502a-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"5162c126-bea7-4999-8f87-b4f03ae8502a\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 11:29:26.955377 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:29:26.955101    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5162c126-bea7-4999-8f87-b4f03ae8502a-config-out\") pod \"alertmanager-main-0\" (UID: \"5162c126-bea7-4999-8f87-b4f03ae8502a\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 11:29:26.955377 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:29:26.955256    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5162c126-bea7-4999-8f87-b4f03ae8502a-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"5162c126-bea7-4999-8f87-b4f03ae8502a\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 11:29:26.955377 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:29:26.955259    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5162c126-bea7-4999-8f87-b4f03ae8502a-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"5162c126-bea7-4999-8f87-b4f03ae8502a\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 11:29:26.955377 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:29:26.955355    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5162c126-bea7-4999-8f87-b4f03ae8502a-tls-assets\") pod \"alertmanager-main-0\" (UID: \"5162c126-bea7-4999-8f87-b4f03ae8502a\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 11:29:26.955664 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:29:26.955528    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5162c126-bea7-4999-8f87-b4f03ae8502a-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"5162c126-bea7-4999-8f87-b4f03ae8502a\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 11:29:26.955664 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:29:26.955612    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5162c126-bea7-4999-8f87-b4f03ae8502a-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"5162c126-bea7-4999-8f87-b4f03ae8502a\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 11:29:26.955664 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:29:26.955650    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5162c126-bea7-4999-8f87-b4f03ae8502a-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"5162c126-bea7-4999-8f87-b4f03ae8502a\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 11:29:26.956309 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:29:26.956284    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5162c126-bea7-4999-8f87-b4f03ae8502a-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"5162c126-bea7-4999-8f87-b4f03ae8502a\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 11:29:26.956550 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:29:26.956530    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5162c126-bea7-4999-8f87-b4f03ae8502a-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"5162c126-bea7-4999-8f87-b4f03ae8502a\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 11:29:26.957316 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:29:26.957299    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5162c126-bea7-4999-8f87-b4f03ae8502a-web-config\") pod \"alertmanager-main-0\" (UID: \"5162c126-bea7-4999-8f87-b4f03ae8502a\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 11:29:26.958093 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:29:26.958070    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5162c126-bea7-4999-8f87-b4f03ae8502a-config-volume\") pod \"alertmanager-main-0\" (UID: \"5162c126-bea7-4999-8f87-b4f03ae8502a\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 11:29:26.965360 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:29:26.965336    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zphvc\" (UniqueName: \"kubernetes.io/projected/5162c126-bea7-4999-8f87-b4f03ae8502a-kube-api-access-zphvc\") pod \"alertmanager-main-0\" (UID: \"5162c126-bea7-4999-8f87-b4f03ae8502a\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 11:29:27.048022 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:29:27.047983    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
May 26 11:29:27.183821 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:29:27.183793    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
May 26 11:29:27.186379 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:29:27.186352    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5162c126_bea7_4999_8f87_b4f03ae8502a.slice/crio-381ac7869f220f974af58a958ae2f072951702133e6b257e39491d0f32c1c5e1 WatchSource:0}: Error finding container 381ac7869f220f974af58a958ae2f072951702133e6b257e39491d0f32c1c5e1: Status 404 returned error can't find the container with id 381ac7869f220f974af58a958ae2f072951702133e6b257e39491d0f32c1c5e1
May 26 11:29:27.671478 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:29:27.671432    2572 generic.go:358] "Generic (PLEG): container finished" podID="5162c126-bea7-4999-8f87-b4f03ae8502a" containerID="d17be4719834d3c27d0ee0e5c287e359972340d1ab29697eb1e4cafa38b3d731" exitCode=0
May 26 11:29:27.675505 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:29:27.675472    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="75506346-244a-4a4c-9d67-26fc614aa68c" path="/var/lib/kubelet/pods/75506346-244a-4a4c-9d67-26fc614aa68c/volumes"
May 26 11:29:27.676013 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:29:27.675994    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5162c126-bea7-4999-8f87-b4f03ae8502a","Type":"ContainerDied","Data":"d17be4719834d3c27d0ee0e5c287e359972340d1ab29697eb1e4cafa38b3d731"}
May 26 11:29:27.676054 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:29:27.676023    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5162c126-bea7-4999-8f87-b4f03ae8502a","Type":"ContainerStarted","Data":"381ac7869f220f974af58a958ae2f072951702133e6b257e39491d0f32c1c5e1"}
May 26 11:29:28.681654 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:29:28.681623    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5162c126-bea7-4999-8f87-b4f03ae8502a","Type":"ContainerStarted","Data":"8bc775fe7c876dbe40dd3d0cbe0105469cb9548c757a8c2442e65361de3acd37"}
May 26 11:29:28.681654 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:29:28.681658    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5162c126-bea7-4999-8f87-b4f03ae8502a","Type":"ContainerStarted","Data":"7da97edb15b878863c816709ac440091f8dbd5aad37fb38ac7ad1a6bf91a2f1a"}
May 26 11:29:28.682092 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:29:28.681667    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5162c126-bea7-4999-8f87-b4f03ae8502a","Type":"ContainerStarted","Data":"0396da11dfaf0a400025418cffcb522f145a31740b55b6f4e67b8995a5189826"}
May 26 11:29:28.682092 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:29:28.681677    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5162c126-bea7-4999-8f87-b4f03ae8502a","Type":"ContainerStarted","Data":"6f16d6076c8d1e4d003ca3a99031aaaf9542b96fddd1ad16da57f7c45a3b366e"}
May 26 11:29:28.682092 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:29:28.681685    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5162c126-bea7-4999-8f87-b4f03ae8502a","Type":"ContainerStarted","Data":"4f58a3958a1649c04609ccfc9e890304e0ca053207470b3b79a572825577256b"}
May 26 11:29:28.682092 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:29:28.681693    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5162c126-bea7-4999-8f87-b4f03ae8502a","Type":"ContainerStarted","Data":"1b1faa55c6f63f7a73540d695c10205b2d55bcae9b3fbc8fb3f6e074d864a4ee"}
May 26 11:29:28.713371 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:29:28.713326    2572 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.713312172 podStartE2EDuration="2.713312172s" podCreationTimestamp="2026-05-26 11:29:26 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 11:29:28.712754714 +0000 UTC m=+291.614965042" watchObservedRunningTime="2026-05-26 11:29:28.713312172 +0000 UTC m=+291.615522533"
May 26 11:29:33.656283 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:29:33.656249    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-wb9kk"
May 26 11:29:37.560475 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:29:37.560426    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-5bfgg_c6373342-dd07-4497-a4f3-978ea9613b6e/console-operator/2.log"
May 26 11:29:37.560850 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:29:37.560593    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-5bfgg_c6373342-dd07-4497-a4f3-978ea9613b6e/console-operator/2.log"
May 26 11:29:37.567709 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:29:37.567683    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-js8cc_7264c907-f536-4719-905a-2bbda9e3c336/ovn-acl-logging/0.log"
May 26 11:29:37.567933 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:29:37.567913    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-js8cc_7264c907-f536-4719-905a-2bbda9e3c336/ovn-acl-logging/0.log"
May 26 11:29:37.570338 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:29:37.570322    2572 kubelet.go:1628] "Image garbage collection succeeded"
May 26 11:34:37.585550 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:34:37.585523    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-5bfgg_c6373342-dd07-4497-a4f3-978ea9613b6e/console-operator/2.log"
May 26 11:34:37.586139 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:34:37.586121    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-5bfgg_c6373342-dd07-4497-a4f3-978ea9613b6e/console-operator/2.log"
May 26 11:34:37.591607 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:34:37.591584    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-js8cc_7264c907-f536-4719-905a-2bbda9e3c336/ovn-acl-logging/0.log"
May 26 11:34:37.592133 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:34:37.592116    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-js8cc_7264c907-f536-4719-905a-2bbda9e3c336/ovn-acl-logging/0.log"
May 26 11:34:50.596213 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:34:50.596147    2572 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-86662"]
May 26 11:34:50.599604 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:34:50.599588    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-86662"
May 26 11:34:50.602685 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:34:50.602666    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
May 26 11:34:50.609580 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:34:50.609562    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
May 26 11:34:50.613752 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:34:50.613736    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
May 26 11:34:50.619768 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:34:50.619753    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"custom-metrics-autoscaler-operator-dockercfg-tdf88\""
May 26 11:34:50.625861 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:34:50.625789    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-86662"]
May 26 11:34:50.664116 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:34:50.664089    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzc4j\" (UniqueName: \"kubernetes.io/projected/ae23b202-3618-4067-bcfa-218b84e27207-kube-api-access-rzc4j\") pod \"custom-metrics-autoscaler-operator-bbf89fd5d-86662\" (UID: \"ae23b202-3618-4067-bcfa-218b84e27207\") " pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-86662"
May 26 11:34:50.664264 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:34:50.664122    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/ae23b202-3618-4067-bcfa-218b84e27207-certificates\") pod \"custom-metrics-autoscaler-operator-bbf89fd5d-86662\" (UID: \"ae23b202-3618-4067-bcfa-218b84e27207\") " pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-86662"
May 26 11:34:50.764713 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:34:50.764683    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-rzc4j\" (UniqueName: \"kubernetes.io/projected/ae23b202-3618-4067-bcfa-218b84e27207-kube-api-access-rzc4j\") pod \"custom-metrics-autoscaler-operator-bbf89fd5d-86662\" (UID: \"ae23b202-3618-4067-bcfa-218b84e27207\") " pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-86662"
May 26 11:34:50.764881 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:34:50.764720    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/ae23b202-3618-4067-bcfa-218b84e27207-certificates\") pod \"custom-metrics-autoscaler-operator-bbf89fd5d-86662\" (UID: \"ae23b202-3618-4067-bcfa-218b84e27207\") " pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-86662"
May 26 11:34:50.766941 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:34:50.766920    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/ae23b202-3618-4067-bcfa-218b84e27207-certificates\") pod \"custom-metrics-autoscaler-operator-bbf89fd5d-86662\" (UID: \"ae23b202-3618-4067-bcfa-218b84e27207\") " pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-86662"
May 26 11:34:50.774528 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:34:50.774504    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzc4j\" (UniqueName: \"kubernetes.io/projected/ae23b202-3618-4067-bcfa-218b84e27207-kube-api-access-rzc4j\") pod \"custom-metrics-autoscaler-operator-bbf89fd5d-86662\" (UID: \"ae23b202-3618-4067-bcfa-218b84e27207\") " pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-86662"
May 26 11:34:50.909426 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:34:50.909398    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-86662"
May 26 11:34:51.042330 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:34:51.042303    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-86662"]
May 26 11:34:51.044955 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:34:51.044923    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae23b202_3618_4067_bcfa_218b84e27207.slice/crio-6029ece16b51c4dfdf9167b0b0cdc183f9b05b94011d9e2f20de82f2c64f6c7d WatchSource:0}: Error finding container 6029ece16b51c4dfdf9167b0b0cdc183f9b05b94011d9e2f20de82f2c64f6c7d: Status 404 returned error can't find the container with id 6029ece16b51c4dfdf9167b0b0cdc183f9b05b94011d9e2f20de82f2c64f6c7d
May 26 11:34:51.046562 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:34:51.046541    2572 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 26 11:34:51.644446 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:34:51.644349    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-86662" event={"ID":"ae23b202-3618-4067-bcfa-218b84e27207","Type":"ContainerStarted","Data":"6029ece16b51c4dfdf9167b0b0cdc183f9b05b94011d9e2f20de82f2c64f6c7d"}
May 26 11:34:54.656731 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:34:54.656695    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-86662" event={"ID":"ae23b202-3618-4067-bcfa-218b84e27207","Type":"ContainerStarted","Data":"77324b60728880dea2f5839873f48e39a7cf6cf59f26ba8c90ca1d164c94f0ca"}
May 26 11:34:54.657256 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:34:54.656811    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-86662"
May 26 11:34:54.680473 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:34:54.678919    2572 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-86662" podStartSLOduration=1.244527291 podStartE2EDuration="4.678900041s" podCreationTimestamp="2026-05-26 11:34:50 +0000 UTC" firstStartedPulling="2026-05-26 11:34:51.046719172 +0000 UTC m=+613.948929499" lastFinishedPulling="2026-05-26 11:34:54.481091939 +0000 UTC m=+617.383302249" observedRunningTime="2026-05-26 11:34:54.676941734 +0000 UTC m=+617.579152062" watchObservedRunningTime="2026-05-26 11:34:54.678900041 +0000 UTC m=+617.581110371"
May 26 11:34:55.018203 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:34:55.018142    2572 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-operator-ffbb595cb-pv6c2"]
May 26 11:34:55.021383 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:34:55.021361    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-ffbb595cb-pv6c2"
May 26 11:34:55.023902 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:34:55.023876    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
May 26 11:34:55.024022 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:34:55.023886    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-mz46t\""
May 26 11:34:55.024022 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:34:55.023924    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-certs\""
May 26 11:34:55.029702 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:34:55.029678    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-ffbb595cb-pv6c2"]
May 26 11:34:55.110473 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:34:55.110431    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/7e970f03-98bc-4803-8984-507cd87fcdeb-cabundle0\") pod \"keda-operator-ffbb595cb-pv6c2\" (UID: \"7e970f03-98bc-4803-8984-507cd87fcdeb\") " pod="openshift-keda/keda-operator-ffbb595cb-pv6c2"
May 26 11:34:55.110669 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:34:55.110484    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/7e970f03-98bc-4803-8984-507cd87fcdeb-certificates\") pod \"keda-operator-ffbb595cb-pv6c2\" (UID: \"7e970f03-98bc-4803-8984-507cd87fcdeb\") " pod="openshift-keda/keda-operator-ffbb595cb-pv6c2"
May 26 11:34:55.110669 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:34:55.110553    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7fm7\" (UniqueName: \"kubernetes.io/projected/7e970f03-98bc-4803-8984-507cd87fcdeb-kube-api-access-m7fm7\") pod \"keda-operator-ffbb595cb-pv6c2\" (UID: \"7e970f03-98bc-4803-8984-507cd87fcdeb\") " pod="openshift-keda/keda-operator-ffbb595cb-pv6c2"
May 26 11:34:55.211816 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:34:55.211779    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/7e970f03-98bc-4803-8984-507cd87fcdeb-cabundle0\") pod \"keda-operator-ffbb595cb-pv6c2\" (UID: \"7e970f03-98bc-4803-8984-507cd87fcdeb\") " pod="openshift-keda/keda-operator-ffbb595cb-pv6c2"
May 26 11:34:55.211988 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:34:55.211826    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/7e970f03-98bc-4803-8984-507cd87fcdeb-certificates\") pod \"keda-operator-ffbb595cb-pv6c2\" (UID: \"7e970f03-98bc-4803-8984-507cd87fcdeb\") " pod="openshift-keda/keda-operator-ffbb595cb-pv6c2"
May 26 11:34:55.211988 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:34:55.211854    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-m7fm7\" (UniqueName: \"kubernetes.io/projected/7e970f03-98bc-4803-8984-507cd87fcdeb-kube-api-access-m7fm7\") pod \"keda-operator-ffbb595cb-pv6c2\" (UID: \"7e970f03-98bc-4803-8984-507cd87fcdeb\") " pod="openshift-keda/keda-operator-ffbb595cb-pv6c2"
May 26 11:34:55.212065 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:34:55.211990    2572 secret.go:281] references non-existent secret key: ca.crt
May 26 11:34:55.212065 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:34:55.212014    2572 projected.go:277] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
May 26 11:34:55.212065 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:34:55.212026    2572 projected.go:194] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-ffbb595cb-pv6c2: references non-existent secret key: ca.crt
May 26 11:34:55.212154 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:34:55.212104    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7e970f03-98bc-4803-8984-507cd87fcdeb-certificates podName:7e970f03-98bc-4803-8984-507cd87fcdeb nodeName:}" failed. No retries permitted until 2026-05-26 11:34:55.712080153 +0000 UTC m=+618.614290477 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/7e970f03-98bc-4803-8984-507cd87fcdeb-certificates") pod "keda-operator-ffbb595cb-pv6c2" (UID: "7e970f03-98bc-4803-8984-507cd87fcdeb") : references non-existent secret key: ca.crt
May 26 11:34:55.212541 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:34:55.212522    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/7e970f03-98bc-4803-8984-507cd87fcdeb-cabundle0\") pod \"keda-operator-ffbb595cb-pv6c2\" (UID: \"7e970f03-98bc-4803-8984-507cd87fcdeb\") " pod="openshift-keda/keda-operator-ffbb595cb-pv6c2"
May 26 11:34:55.225247 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:34:55.225227    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7fm7\" (UniqueName: \"kubernetes.io/projected/7e970f03-98bc-4803-8984-507cd87fcdeb-kube-api-access-m7fm7\") pod \"keda-operator-ffbb595cb-pv6c2\" (UID: \"7e970f03-98bc-4803-8984-507cd87fcdeb\") " pod="openshift-keda/keda-operator-ffbb595cb-pv6c2"
May 26 11:34:55.396667 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:34:55.396565    2572 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-metrics-apiserver-7c9f485588-4j8qj"]
May 26 11:34:55.400240 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:34:55.400219    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-7c9f485588-4j8qj"
May 26 11:34:55.402894 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:34:55.402875    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-metrics-apiserver-certs\""
May 26 11:34:55.409823 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:34:55.409800    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-7c9f485588-4j8qj"]
May 26 11:34:55.514448 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:34:55.514406    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlcgl\" (UniqueName: \"kubernetes.io/projected/9168b497-67e3-4194-8fae-40cd09f75734-kube-api-access-wlcgl\") pod \"keda-metrics-apiserver-7c9f485588-4j8qj\" (UID: \"9168b497-67e3-4194-8fae-40cd09f75734\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-4j8qj"
May 26 11:34:55.514621 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:34:55.514466    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/9168b497-67e3-4194-8fae-40cd09f75734-temp-vol\") pod \"keda-metrics-apiserver-7c9f485588-4j8qj\" (UID: \"9168b497-67e3-4194-8fae-40cd09f75734\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-4j8qj"
May 26 11:34:55.514621 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:34:55.514505    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/9168b497-67e3-4194-8fae-40cd09f75734-certificates\") pod \"keda-metrics-apiserver-7c9f485588-4j8qj\" (UID: \"9168b497-67e3-4194-8fae-40cd09f75734\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-4j8qj"
May 26 11:34:55.615348 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:34:55.615317    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/9168b497-67e3-4194-8fae-40cd09f75734-temp-vol\") pod \"keda-metrics-apiserver-7c9f485588-4j8qj\" (UID: \"9168b497-67e3-4194-8fae-40cd09f75734\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-4j8qj"
May 26 11:34:55.615348 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:34:55.615358    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/9168b497-67e3-4194-8fae-40cd09f75734-certificates\") pod \"keda-metrics-apiserver-7c9f485588-4j8qj\" (UID: \"9168b497-67e3-4194-8fae-40cd09f75734\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-4j8qj"
May 26 11:34:55.615565 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:34:55.615419    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-wlcgl\" (UniqueName: \"kubernetes.io/projected/9168b497-67e3-4194-8fae-40cd09f75734-kube-api-access-wlcgl\") pod \"keda-metrics-apiserver-7c9f485588-4j8qj\" (UID: \"9168b497-67e3-4194-8fae-40cd09f75734\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-4j8qj"
May 26 11:34:55.615565 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:34:55.615529    2572 secret.go:281] references non-existent secret key: tls.crt
May 26 11:34:55.615565 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:34:55.615548    2572 projected.go:277] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: tls.crt
May 26 11:34:55.615565 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:34:55.615565    2572 projected.go:264] Couldn't get secret openshift-keda/keda-metrics-apiserver-certs: secret "keda-metrics-apiserver-certs" not found
May 26 11:34:55.615705 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:34:55.615581    2572 projected.go:194] Error preparing data for projected volume certificates for pod openshift-keda/keda-metrics-apiserver-7c9f485588-4j8qj: [references non-existent secret key: tls.crt, secret "keda-metrics-apiserver-certs" not found]
May 26 11:34:55.615705 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:34:55.615627    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9168b497-67e3-4194-8fae-40cd09f75734-certificates podName:9168b497-67e3-4194-8fae-40cd09f75734 nodeName:}" failed. No retries permitted until 2026-05-26 11:34:56.115611924 +0000 UTC m=+619.017822231 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/9168b497-67e3-4194-8fae-40cd09f75734-certificates") pod "keda-metrics-apiserver-7c9f485588-4j8qj" (UID: "9168b497-67e3-4194-8fae-40cd09f75734") : [references non-existent secret key: tls.crt, secret "keda-metrics-apiserver-certs" not found]
May 26 11:34:55.615705 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:34:55.615681    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/9168b497-67e3-4194-8fae-40cd09f75734-temp-vol\") pod \"keda-metrics-apiserver-7c9f485588-4j8qj\" (UID: \"9168b497-67e3-4194-8fae-40cd09f75734\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-4j8qj"
May 26 11:34:55.631313 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:34:55.631289    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlcgl\" (UniqueName: \"kubernetes.io/projected/9168b497-67e3-4194-8fae-40cd09f75734-kube-api-access-wlcgl\") pod \"keda-metrics-apiserver-7c9f485588-4j8qj\" (UID: \"9168b497-67e3-4194-8fae-40cd09f75734\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-4j8qj"
May 26 11:34:55.715843 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:34:55.715813    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/7e970f03-98bc-4803-8984-507cd87fcdeb-certificates\") pod \"keda-operator-ffbb595cb-pv6c2\" (UID: \"7e970f03-98bc-4803-8984-507cd87fcdeb\") " pod="openshift-keda/keda-operator-ffbb595cb-pv6c2"
May 26 11:34:55.716226 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:34:55.715921    2572 secret.go:281] references non-existent secret key: ca.crt
May 26 11:34:55.716226 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:34:55.715939    2572 projected.go:277] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
May 26 11:34:55.716226 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:34:55.715948    2572 projected.go:194] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-ffbb595cb-pv6c2: references non-existent secret key: ca.crt
May 26 11:34:55.716226 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:34:55.716003    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7e970f03-98bc-4803-8984-507cd87fcdeb-certificates podName:7e970f03-98bc-4803-8984-507cd87fcdeb nodeName:}" failed. No retries permitted until 2026-05-26 11:34:56.715987281 +0000 UTC m=+619.618197593 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/7e970f03-98bc-4803-8984-507cd87fcdeb-certificates") pod "keda-operator-ffbb595cb-pv6c2" (UID: "7e970f03-98bc-4803-8984-507cd87fcdeb") : references non-existent secret key: ca.crt
May 26 11:34:56.119561 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:34:56.119472    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/9168b497-67e3-4194-8fae-40cd09f75734-certificates\") pod \"keda-metrics-apiserver-7c9f485588-4j8qj\" (UID: \"9168b497-67e3-4194-8fae-40cd09f75734\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-4j8qj"
May 26 11:34:56.119712 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:34:56.119634    2572 secret.go:281] references non-existent secret key: tls.crt
May 26 11:34:56.119712 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:34:56.119656    2572 projected.go:277] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: tls.crt
May 26 11:34:56.119712 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:34:56.119676    2572 projected.go:194] Error preparing data for projected volume certificates for pod openshift-keda/keda-metrics-apiserver-7c9f485588-4j8qj: references non-existent secret key: tls.crt
May 26 11:34:56.119814 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:34:56.119752    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9168b497-67e3-4194-8fae-40cd09f75734-certificates podName:9168b497-67e3-4194-8fae-40cd09f75734 nodeName:}" failed. No retries permitted until 2026-05-26 11:34:57.119735245 +0000 UTC m=+620.021945554 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/9168b497-67e3-4194-8fae-40cd09f75734-certificates") pod "keda-metrics-apiserver-7c9f485588-4j8qj" (UID: "9168b497-67e3-4194-8fae-40cd09f75734") : references non-existent secret key: tls.crt
May 26 11:34:56.724420 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:34:56.724376    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/7e970f03-98bc-4803-8984-507cd87fcdeb-certificates\") pod \"keda-operator-ffbb595cb-pv6c2\" (UID: \"7e970f03-98bc-4803-8984-507cd87fcdeb\") " pod="openshift-keda/keda-operator-ffbb595cb-pv6c2"
May 26 11:34:56.724787 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:34:56.724499    2572 secret.go:281] references non-existent secret key: ca.crt
May 26 11:34:56.724787 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:34:56.724511    2572 projected.go:277] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
May 26 11:34:56.724787 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:34:56.724519    2572 projected.go:194] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-ffbb595cb-pv6c2: references non-existent secret key: ca.crt
May 26 11:34:56.724787 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:34:56.724571    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7e970f03-98bc-4803-8984-507cd87fcdeb-certificates podName:7e970f03-98bc-4803-8984-507cd87fcdeb nodeName:}" failed. No retries permitted until 2026-05-26 11:34:58.724553861 +0000 UTC m=+621.626764170 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/7e970f03-98bc-4803-8984-507cd87fcdeb-certificates") pod "keda-operator-ffbb595cb-pv6c2" (UID: "7e970f03-98bc-4803-8984-507cd87fcdeb") : references non-existent secret key: ca.crt
May 26 11:34:57.128941 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:34:57.128836    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/9168b497-67e3-4194-8fae-40cd09f75734-certificates\") pod \"keda-metrics-apiserver-7c9f485588-4j8qj\" (UID: \"9168b497-67e3-4194-8fae-40cd09f75734\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-4j8qj"
May 26 11:34:57.129108 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:34:57.128982    2572 secret.go:281] references non-existent secret key: tls.crt
May 26 11:34:57.129108 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:34:57.129003    2572 projected.go:277] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: tls.crt
May 26 11:34:57.129108 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:34:57.129023    2572 projected.go:194] Error preparing data for projected volume certificates for pod openshift-keda/keda-metrics-apiserver-7c9f485588-4j8qj: references non-existent secret key: tls.crt
May 26 11:34:57.129108 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:34:57.129078    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9168b497-67e3-4194-8fae-40cd09f75734-certificates podName:9168b497-67e3-4194-8fae-40cd09f75734 nodeName:}" failed. No retries permitted until 2026-05-26 11:34:59.129063627 +0000 UTC m=+622.031273937 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/9168b497-67e3-4194-8fae-40cd09f75734-certificates") pod "keda-metrics-apiserver-7c9f485588-4j8qj" (UID: "9168b497-67e3-4194-8fae-40cd09f75734") : references non-existent secret key: tls.crt
May 26 11:34:58.742904 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:34:58.742868    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/7e970f03-98bc-4803-8984-507cd87fcdeb-certificates\") pod \"keda-operator-ffbb595cb-pv6c2\" (UID: \"7e970f03-98bc-4803-8984-507cd87fcdeb\") " pod="openshift-keda/keda-operator-ffbb595cb-pv6c2"
May 26 11:34:58.743320 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:34:58.743039    2572 secret.go:281] references non-existent secret key: ca.crt
May 26 11:34:58.743320 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:34:58.743064    2572 projected.go:277] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
May 26 11:34:58.743320 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:34:58.743076    2572 projected.go:194] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-ffbb595cb-pv6c2: references non-existent secret key: ca.crt
May 26 11:34:58.743320 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:34:58.743153    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7e970f03-98bc-4803-8984-507cd87fcdeb-certificates podName:7e970f03-98bc-4803-8984-507cd87fcdeb nodeName:}" failed. No retries permitted until 2026-05-26 11:35:02.743132665 +0000 UTC m=+625.645342982 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/7e970f03-98bc-4803-8984-507cd87fcdeb-certificates") pod "keda-operator-ffbb595cb-pv6c2" (UID: "7e970f03-98bc-4803-8984-507cd87fcdeb") : references non-existent secret key: ca.crt
May 26 11:34:59.146545 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:34:59.146434    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/9168b497-67e3-4194-8fae-40cd09f75734-certificates\") pod \"keda-metrics-apiserver-7c9f485588-4j8qj\" (UID: \"9168b497-67e3-4194-8fae-40cd09f75734\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-4j8qj"
May 26 11:34:59.146724 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:34:59.146588    2572 secret.go:281] references non-existent secret key: tls.crt
May 26 11:34:59.146724 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:34:59.146610    2572 projected.go:277] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: tls.crt
May 26 11:34:59.146724 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:34:59.146629    2572 projected.go:194] Error preparing data for projected volume certificates for pod openshift-keda/keda-metrics-apiserver-7c9f485588-4j8qj: references non-existent secret key: tls.crt
May 26 11:34:59.146724 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:34:59.146682    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9168b497-67e3-4194-8fae-40cd09f75734-certificates podName:9168b497-67e3-4194-8fae-40cd09f75734 nodeName:}" failed. No retries permitted until 2026-05-26 11:35:03.146668259 +0000 UTC m=+626.048878566 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/9168b497-67e3-4194-8fae-40cd09f75734-certificates") pod "keda-metrics-apiserver-7c9f485588-4j8qj" (UID: "9168b497-67e3-4194-8fae-40cd09f75734") : references non-existent secret key: tls.crt
May 26 11:35:02.779004 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:35:02.778962    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/7e970f03-98bc-4803-8984-507cd87fcdeb-certificates\") pod \"keda-operator-ffbb595cb-pv6c2\" (UID: \"7e970f03-98bc-4803-8984-507cd87fcdeb\") " pod="openshift-keda/keda-operator-ffbb595cb-pv6c2"
May 26 11:35:02.781534 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:35:02.781508    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/7e970f03-98bc-4803-8984-507cd87fcdeb-certificates\") pod \"keda-operator-ffbb595cb-pv6c2\" (UID: \"7e970f03-98bc-4803-8984-507cd87fcdeb\") " pod="openshift-keda/keda-operator-ffbb595cb-pv6c2"
May 26 11:35:02.832574 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:35:02.832534    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-ffbb595cb-pv6c2"
May 26 11:35:02.959068 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:35:02.959041    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-ffbb595cb-pv6c2"]
May 26 11:35:02.961435 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:35:02.961410    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e970f03_98bc_4803_8984_507cd87fcdeb.slice/crio-99f4106113e12a2d0d4e695aa07e7e99a494ec04f817c3cf0edaf26b2b3e16bf WatchSource:0}: Error finding container 99f4106113e12a2d0d4e695aa07e7e99a494ec04f817c3cf0edaf26b2b3e16bf: Status 404 returned error can't find the container with id 99f4106113e12a2d0d4e695aa07e7e99a494ec04f817c3cf0edaf26b2b3e16bf
May 26 11:35:03.182345 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:35:03.182307    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/9168b497-67e3-4194-8fae-40cd09f75734-certificates\") pod \"keda-metrics-apiserver-7c9f485588-4j8qj\" (UID: \"9168b497-67e3-4194-8fae-40cd09f75734\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-4j8qj"
May 26 11:35:03.184774 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:35:03.184745    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/9168b497-67e3-4194-8fae-40cd09f75734-certificates\") pod \"keda-metrics-apiserver-7c9f485588-4j8qj\" (UID: \"9168b497-67e3-4194-8fae-40cd09f75734\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-4j8qj"
May 26 11:35:03.211555 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:35:03.211527    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-7c9f485588-4j8qj"
May 26 11:35:03.336452 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:35:03.336428    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-7c9f485588-4j8qj"]
May 26 11:35:03.691820 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:35:03.691781    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-7c9f485588-4j8qj" event={"ID":"9168b497-67e3-4194-8fae-40cd09f75734","Type":"ContainerStarted","Data":"af7cdf3257a8d6161ddac0cc2866f9341f6db876f4068881659929cce10eae55"}
May 26 11:35:03.692581 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:35:03.692558    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-ffbb595cb-pv6c2" event={"ID":"7e970f03-98bc-4803-8984-507cd87fcdeb","Type":"ContainerStarted","Data":"99f4106113e12a2d0d4e695aa07e7e99a494ec04f817c3cf0edaf26b2b3e16bf"}
May 26 11:35:07.707834 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:35:07.707794    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-ffbb595cb-pv6c2" event={"ID":"7e970f03-98bc-4803-8984-507cd87fcdeb","Type":"ContainerStarted","Data":"96b39aa49d768591134fb3a92371c8440342061552d9a6dd2db531cf2ca085ab"}
May 26 11:35:07.708352 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:35:07.707890    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-operator-ffbb595cb-pv6c2"
May 26 11:35:07.709213 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:35:07.709164    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-7c9f485588-4j8qj" event={"ID":"9168b497-67e3-4194-8fae-40cd09f75734","Type":"ContainerStarted","Data":"1b26a800ffff42f945b1c5d870c52da346bd359a12ab8427faefae5c0cfaac5a"}
May 26 11:35:07.709337 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:35:07.709254    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-metrics-apiserver-7c9f485588-4j8qj"
May 26 11:35:07.725813 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:35:07.725769    2572 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-keda/keda-operator-ffbb595cb-pv6c2" podStartSLOduration=8.500698869 podStartE2EDuration="12.725757853s" podCreationTimestamp="2026-05-26 11:34:55 +0000 UTC" firstStartedPulling="2026-05-26 11:35:02.963040662 +0000 UTC m=+625.865250970" lastFinishedPulling="2026-05-26 11:35:07.188099646 +0000 UTC m=+630.090309954" observedRunningTime="2026-05-26 11:35:07.724410891 +0000 UTC m=+630.626621231" watchObservedRunningTime="2026-05-26 11:35:07.725757853 +0000 UTC m=+630.627968181"
May 26 11:35:07.740893 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:35:07.740848    2572 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-keda/keda-metrics-apiserver-7c9f485588-4j8qj" podStartSLOduration=8.898647205 podStartE2EDuration="12.740833145s" podCreationTimestamp="2026-05-26 11:34:55 +0000 UTC" firstStartedPulling="2026-05-26 11:35:03.339929968 +0000 UTC m=+626.242140276" lastFinishedPulling="2026-05-26 11:35:07.182115905 +0000 UTC m=+630.084326216" observedRunningTime="2026-05-26 11:35:07.739676518 +0000 UTC m=+630.641886849" watchObservedRunningTime="2026-05-26 11:35:07.740833145 +0000 UTC m=+630.643043476"
May 26 11:35:15.663318 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:35:15.663289    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-86662"
May 26 11:35:18.718146 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:35:18.718111    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-metrics-apiserver-7c9f485588-4j8qj"
May 26 11:35:28.716077 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:35:28.716049    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-operator-ffbb595cb-pv6c2"
May 26 11:36:11.307459 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:36:11.307425    2572 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-68b757865b-rz24t"]
May 26 11:36:11.309694 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:36:11.309678    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-68b757865b-rz24t"
May 26 11:36:11.315485 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:36:11.315464    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
May 26 11:36:11.316696 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:36:11.316676    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
May 26 11:36:11.316778 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:36:11.316676    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-l94k4\""
May 26 11:36:11.327658 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:36:11.327634    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-68b757865b-rz24t"]
May 26 11:36:11.346843 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:36:11.346812    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f4ae79aa-0352-4858-976c-4b6cfcc50f70-bound-sa-token\") pod \"cert-manager-cainjector-68b757865b-rz24t\" (UID: \"f4ae79aa-0352-4858-976c-4b6cfcc50f70\") " pod="cert-manager/cert-manager-cainjector-68b757865b-rz24t"
May 26 11:36:11.346965 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:36:11.346880    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gld9g\" (UniqueName: \"kubernetes.io/projected/f4ae79aa-0352-4858-976c-4b6cfcc50f70-kube-api-access-gld9g\") pod \"cert-manager-cainjector-68b757865b-rz24t\" (UID: \"f4ae79aa-0352-4858-976c-4b6cfcc50f70\") " pod="cert-manager/cert-manager-cainjector-68b757865b-rz24t"
May 26 11:36:11.447594 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:36:11.447558    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f4ae79aa-0352-4858-976c-4b6cfcc50f70-bound-sa-token\") pod \"cert-manager-cainjector-68b757865b-rz24t\" (UID: \"f4ae79aa-0352-4858-976c-4b6cfcc50f70\") " pod="cert-manager/cert-manager-cainjector-68b757865b-rz24t"
May 26 11:36:11.447768 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:36:11.447635    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-gld9g\" (UniqueName: \"kubernetes.io/projected/f4ae79aa-0352-4858-976c-4b6cfcc50f70-kube-api-access-gld9g\") pod \"cert-manager-cainjector-68b757865b-rz24t\" (UID: \"f4ae79aa-0352-4858-976c-4b6cfcc50f70\") " pod="cert-manager/cert-manager-cainjector-68b757865b-rz24t"
May 26 11:36:11.469881 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:36:11.469854    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gld9g\" (UniqueName: \"kubernetes.io/projected/f4ae79aa-0352-4858-976c-4b6cfcc50f70-kube-api-access-gld9g\") pod \"cert-manager-cainjector-68b757865b-rz24t\" (UID: \"f4ae79aa-0352-4858-976c-4b6cfcc50f70\") " pod="cert-manager/cert-manager-cainjector-68b757865b-rz24t"
May 26 11:36:11.476628 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:36:11.476601    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f4ae79aa-0352-4858-976c-4b6cfcc50f70-bound-sa-token\") pod \"cert-manager-cainjector-68b757865b-rz24t\" (UID: \"f4ae79aa-0352-4858-976c-4b6cfcc50f70\") " pod="cert-manager/cert-manager-cainjector-68b757865b-rz24t"
May 26 11:36:11.635084 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:36:11.634996    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-68b757865b-rz24t"
May 26 11:36:11.781637 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:36:11.781604    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-68b757865b-rz24t"]
May 26 11:36:11.785006 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:36:11.784976    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf4ae79aa_0352_4858_976c_4b6cfcc50f70.slice/crio-117f0b5bd8649638809757c9e21b573da8db03c973a9aff3e84c8c615d364b57 WatchSource:0}: Error finding container 117f0b5bd8649638809757c9e21b573da8db03c973a9aff3e84c8c615d364b57: Status 404 returned error can't find the container with id 117f0b5bd8649638809757c9e21b573da8db03c973a9aff3e84c8c615d364b57
May 26 11:36:11.916282 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:36:11.916250    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-68b757865b-rz24t" event={"ID":"f4ae79aa-0352-4858-976c-4b6cfcc50f70","Type":"ContainerStarted","Data":"117f0b5bd8649638809757c9e21b573da8db03c973a9aff3e84c8c615d364b57"}
May 26 11:36:15.934213 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:36:15.934164    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-68b757865b-rz24t" event={"ID":"f4ae79aa-0352-4858-976c-4b6cfcc50f70","Type":"ContainerStarted","Data":"3990d6de3edf2060d792af66f71fe1b75c440ec9a6e98ebd3d6744ecc359cdde"}
May 26 11:36:15.990639 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:36:15.990564    2572 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-68b757865b-rz24t" podStartSLOduration=1.309694208 podStartE2EDuration="4.990549089s" podCreationTimestamp="2026-05-26 11:36:11 +0000 UTC" firstStartedPulling="2026-05-26 11:36:11.786915536 +0000 UTC m=+694.689125846" lastFinishedPulling="2026-05-26 11:36:15.467770407 +0000 UTC m=+698.369980727" observedRunningTime="2026-05-26 11:36:15.984227118 +0000 UTC m=+698.886437447" watchObservedRunningTime="2026-05-26 11:36:15.990549089 +0000 UTC m=+698.892759419"
May 26 11:36:42.043987 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:36:42.043896    2572 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-5b8db67594-xhs6k"]
May 26 11:36:42.047462 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:36:42.047440    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5b8db67594-xhs6k"
May 26 11:36:42.052854 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:36:42.052826    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
May 26 11:36:42.053808 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:36:42.053732    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
May 26 11:36:42.053808 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:36:42.053758    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
May 26 11:36:42.054112 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:36:42.054090    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
May 26 11:36:42.054242 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:36:42.054219    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
May 26 11:36:42.054323 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:36:42.054293    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-9rt5d\""
May 26 11:36:42.089834 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:36:42.089801    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5b8db67594-xhs6k"]
May 26 11:36:42.225560 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:36:42.225521    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a7be51b3-3758-416e-bc26-6fdc50383ce2-cert\") pod \"lws-controller-manager-5b8db67594-xhs6k\" (UID: \"a7be51b3-3758-416e-bc26-6fdc50383ce2\") " pod="openshift-lws-operator/lws-controller-manager-5b8db67594-xhs6k"
May 26 11:36:42.225741 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:36:42.225579    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5sqgg\" (UniqueName: \"kubernetes.io/projected/a7be51b3-3758-416e-bc26-6fdc50383ce2-kube-api-access-5sqgg\") pod \"lws-controller-manager-5b8db67594-xhs6k\" (UID: \"a7be51b3-3758-416e-bc26-6fdc50383ce2\") " pod="openshift-lws-operator/lws-controller-manager-5b8db67594-xhs6k"
May 26 11:36:42.225741 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:36:42.225676    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/a7be51b3-3758-416e-bc26-6fdc50383ce2-metrics-cert\") pod \"lws-controller-manager-5b8db67594-xhs6k\" (UID: \"a7be51b3-3758-416e-bc26-6fdc50383ce2\") " pod="openshift-lws-operator/lws-controller-manager-5b8db67594-xhs6k"
May 26 11:36:42.225741 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:36:42.225722    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/a7be51b3-3758-416e-bc26-6fdc50383ce2-manager-config\") pod \"lws-controller-manager-5b8db67594-xhs6k\" (UID: \"a7be51b3-3758-416e-bc26-6fdc50383ce2\") " pod="openshift-lws-operator/lws-controller-manager-5b8db67594-xhs6k"
May 26 11:36:42.326433 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:36:42.326351    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/a7be51b3-3758-416e-bc26-6fdc50383ce2-manager-config\") pod \"lws-controller-manager-5b8db67594-xhs6k\" (UID: \"a7be51b3-3758-416e-bc26-6fdc50383ce2\") " pod="openshift-lws-operator/lws-controller-manager-5b8db67594-xhs6k"
May 26 11:36:42.326433 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:36:42.326421    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a7be51b3-3758-416e-bc26-6fdc50383ce2-cert\") pod \"lws-controller-manager-5b8db67594-xhs6k\" (UID: \"a7be51b3-3758-416e-bc26-6fdc50383ce2\") " pod="openshift-lws-operator/lws-controller-manager-5b8db67594-xhs6k"
May 26 11:36:42.326613 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:36:42.326451    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-5sqgg\" (UniqueName: \"kubernetes.io/projected/a7be51b3-3758-416e-bc26-6fdc50383ce2-kube-api-access-5sqgg\") pod \"lws-controller-manager-5b8db67594-xhs6k\" (UID: \"a7be51b3-3758-416e-bc26-6fdc50383ce2\") " pod="openshift-lws-operator/lws-controller-manager-5b8db67594-xhs6k"
May 26 11:36:42.326613 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:36:42.326484    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/a7be51b3-3758-416e-bc26-6fdc50383ce2-metrics-cert\") pod \"lws-controller-manager-5b8db67594-xhs6k\" (UID: \"a7be51b3-3758-416e-bc26-6fdc50383ce2\") " pod="openshift-lws-operator/lws-controller-manager-5b8db67594-xhs6k"
May 26 11:36:42.327006 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:36:42.326976    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/a7be51b3-3758-416e-bc26-6fdc50383ce2-manager-config\") pod \"lws-controller-manager-5b8db67594-xhs6k\" (UID: \"a7be51b3-3758-416e-bc26-6fdc50383ce2\") " pod="openshift-lws-operator/lws-controller-manager-5b8db67594-xhs6k"
May 26 11:36:42.328870 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:36:42.328851    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a7be51b3-3758-416e-bc26-6fdc50383ce2-cert\") pod \"lws-controller-manager-5b8db67594-xhs6k\" (UID: \"a7be51b3-3758-416e-bc26-6fdc50383ce2\") " pod="openshift-lws-operator/lws-controller-manager-5b8db67594-xhs6k"
May 26 11:36:42.328987 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:36:42.328896    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/a7be51b3-3758-416e-bc26-6fdc50383ce2-metrics-cert\") pod \"lws-controller-manager-5b8db67594-xhs6k\" (UID: \"a7be51b3-3758-416e-bc26-6fdc50383ce2\") " pod="openshift-lws-operator/lws-controller-manager-5b8db67594-xhs6k"
May 26 11:36:42.337405 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:36:42.337378    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5sqgg\" (UniqueName: \"kubernetes.io/projected/a7be51b3-3758-416e-bc26-6fdc50383ce2-kube-api-access-5sqgg\") pod \"lws-controller-manager-5b8db67594-xhs6k\" (UID: \"a7be51b3-3758-416e-bc26-6fdc50383ce2\") " pod="openshift-lws-operator/lws-controller-manager-5b8db67594-xhs6k"
May 26 11:36:42.356208 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:36:42.356164    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5b8db67594-xhs6k"
May 26 11:36:42.495868 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:36:42.495843    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5b8db67594-xhs6k"]
May 26 11:36:42.498182 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:36:42.498154    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7be51b3_3758_416e_bc26_6fdc50383ce2.slice/crio-87a4413c6f2d4a0783c53778ebae95b8aa44133b654875add1f06b799a06c5f9 WatchSource:0}: Error finding container 87a4413c6f2d4a0783c53778ebae95b8aa44133b654875add1f06b799a06c5f9: Status 404 returned error can't find the container with id 87a4413c6f2d4a0783c53778ebae95b8aa44133b654875add1f06b799a06c5f9
May 26 11:36:43.024285 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:36:43.024247    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5b8db67594-xhs6k" event={"ID":"a7be51b3-3758-416e-bc26-6fdc50383ce2","Type":"ContainerStarted","Data":"87a4413c6f2d4a0783c53778ebae95b8aa44133b654875add1f06b799a06c5f9"}
May 26 11:36:46.035325 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:36:46.035282    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5b8db67594-xhs6k" event={"ID":"a7be51b3-3758-416e-bc26-6fdc50383ce2","Type":"ContainerStarted","Data":"e28b3db2dd7eda3dd6ecb6986df24744080212abb2c4448860e7f79f0d6d3ed5"}
May 26 11:36:46.035325 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:36:46.035328    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-5b8db67594-xhs6k"
May 26 11:36:46.052903 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:36:46.052849    2572 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-5b8db67594-xhs6k" podStartSLOduration=1.500579469 podStartE2EDuration="4.052836212s" podCreationTimestamp="2026-05-26 11:36:42 +0000 UTC" firstStartedPulling="2026-05-26 11:36:42.50016872 +0000 UTC m=+725.402379028" lastFinishedPulling="2026-05-26 11:36:45.052425455 +0000 UTC m=+727.954635771" observedRunningTime="2026-05-26 11:36:46.050839231 +0000 UTC m=+728.953049559" watchObservedRunningTime="2026-05-26 11:36:46.052836212 +0000 UTC m=+728.955046540"
May 26 11:36:57.041145 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:36:57.041113    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-5b8db67594-xhs6k"
May 26 11:37:34.377032 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:37:34.376957    2572 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-6c886788f8-9dlsj"]
May 26 11:37:34.380646 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:37:34.380622    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-6c886788f8-9dlsj"
May 26 11:37:34.383851 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:37:34.383832    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
May 26 11:37:34.384125 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:37:34.384106    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
May 26 11:37:34.384248 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:37:34.384233    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
May 26 11:37:34.384988 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:37:34.384973    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
May 26 11:37:34.385042 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:37:34.384979    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-qm7b7\""
May 26 11:37:34.398242 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:37:34.398219    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-6c886788f8-9dlsj"]
May 26 11:37:34.437964 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:37:34.437924    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/b8db013d-69c4-45c8-ad99-468363e53491-plugin-serving-cert\") pod \"kuadrant-console-plugin-6c886788f8-9dlsj\" (UID: \"b8db013d-69c4-45c8-ad99-468363e53491\") " pod="kuadrant-system/kuadrant-console-plugin-6c886788f8-9dlsj"
May 26 11:37:34.438149 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:37:34.438041    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpd8g\" (UniqueName: \"kubernetes.io/projected/b8db013d-69c4-45c8-ad99-468363e53491-kube-api-access-jpd8g\") pod \"kuadrant-console-plugin-6c886788f8-9dlsj\" (UID: \"b8db013d-69c4-45c8-ad99-468363e53491\") " pod="kuadrant-system/kuadrant-console-plugin-6c886788f8-9dlsj"
May 26 11:37:34.438149 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:37:34.438102    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/b8db013d-69c4-45c8-ad99-468363e53491-nginx-conf\") pod \"kuadrant-console-plugin-6c886788f8-9dlsj\" (UID: \"b8db013d-69c4-45c8-ad99-468363e53491\") " pod="kuadrant-system/kuadrant-console-plugin-6c886788f8-9dlsj"
May 26 11:37:34.539360 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:37:34.539324    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-jpd8g\" (UniqueName: \"kubernetes.io/projected/b8db013d-69c4-45c8-ad99-468363e53491-kube-api-access-jpd8g\") pod \"kuadrant-console-plugin-6c886788f8-9dlsj\" (UID: \"b8db013d-69c4-45c8-ad99-468363e53491\") " pod="kuadrant-system/kuadrant-console-plugin-6c886788f8-9dlsj"
May 26 11:37:34.539544 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:37:34.539378    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/b8db013d-69c4-45c8-ad99-468363e53491-nginx-conf\") pod \"kuadrant-console-plugin-6c886788f8-9dlsj\" (UID: \"b8db013d-69c4-45c8-ad99-468363e53491\") " pod="kuadrant-system/kuadrant-console-plugin-6c886788f8-9dlsj"
May 26 11:37:34.539544 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:37:34.539409    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/b8db013d-69c4-45c8-ad99-468363e53491-plugin-serving-cert\") pod \"kuadrant-console-plugin-6c886788f8-9dlsj\" (UID: \"b8db013d-69c4-45c8-ad99-468363e53491\") " pod="kuadrant-system/kuadrant-console-plugin-6c886788f8-9dlsj"
May 26 11:37:34.539544 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:37:34.539512    2572 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found
May 26 11:37:34.539671 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:37:34.539563    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b8db013d-69c4-45c8-ad99-468363e53491-plugin-serving-cert podName:b8db013d-69c4-45c8-ad99-468363e53491 nodeName:}" failed. No retries permitted until 2026-05-26 11:37:35.039548804 +0000 UTC m=+777.941759111 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/b8db013d-69c4-45c8-ad99-468363e53491-plugin-serving-cert") pod "kuadrant-console-plugin-6c886788f8-9dlsj" (UID: "b8db013d-69c4-45c8-ad99-468363e53491") : secret "plugin-serving-cert" not found
May 26 11:37:34.540072 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:37:34.540052    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/b8db013d-69c4-45c8-ad99-468363e53491-nginx-conf\") pod \"kuadrant-console-plugin-6c886788f8-9dlsj\" (UID: \"b8db013d-69c4-45c8-ad99-468363e53491\") " pod="kuadrant-system/kuadrant-console-plugin-6c886788f8-9dlsj"
May 26 11:37:34.548569 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:37:34.548542    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpd8g\" (UniqueName: \"kubernetes.io/projected/b8db013d-69c4-45c8-ad99-468363e53491-kube-api-access-jpd8g\") pod \"kuadrant-console-plugin-6c886788f8-9dlsj\" (UID: \"b8db013d-69c4-45c8-ad99-468363e53491\") " pod="kuadrant-system/kuadrant-console-plugin-6c886788f8-9dlsj"
May 26 11:37:35.044311 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:37:35.044259    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/b8db013d-69c4-45c8-ad99-468363e53491-plugin-serving-cert\") pod \"kuadrant-console-plugin-6c886788f8-9dlsj\" (UID: \"b8db013d-69c4-45c8-ad99-468363e53491\") " pod="kuadrant-system/kuadrant-console-plugin-6c886788f8-9dlsj"
May 26 11:37:35.046748 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:37:35.046713    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/b8db013d-69c4-45c8-ad99-468363e53491-plugin-serving-cert\") pod \"kuadrant-console-plugin-6c886788f8-9dlsj\" (UID: \"b8db013d-69c4-45c8-ad99-468363e53491\") " pod="kuadrant-system/kuadrant-console-plugin-6c886788f8-9dlsj"
May 26 11:37:35.290160 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:37:35.290121    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-6c886788f8-9dlsj"
May 26 11:37:35.421146 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:37:35.421117    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-6c886788f8-9dlsj"]
May 26 11:37:35.421761 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:37:35.421725    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb8db013d_69c4_45c8_ad99_468363e53491.slice/crio-07882da3598d31c0da5965b8167ec7b879d9619cc88888d375abccf721f5ce7f WatchSource:0}: Error finding container 07882da3598d31c0da5965b8167ec7b879d9619cc88888d375abccf721f5ce7f: Status 404 returned error can't find the container with id 07882da3598d31c0da5965b8167ec7b879d9619cc88888d375abccf721f5ce7f
May 26 11:37:36.212420 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:37:36.212379    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-6c886788f8-9dlsj" event={"ID":"b8db013d-69c4-45c8-ad99-468363e53491","Type":"ContainerStarted","Data":"07882da3598d31c0da5965b8167ec7b879d9619cc88888d375abccf721f5ce7f"}
May 26 11:37:40.232224 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:37:40.232114    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-6c886788f8-9dlsj" event={"ID":"b8db013d-69c4-45c8-ad99-468363e53491","Type":"ContainerStarted","Data":"30ae432891353da5cfbe24018063afeae60049b62746c726d6dbde1db326a604"}
May 26 11:37:40.257486 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:37:40.257434    2572 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-6c886788f8-9dlsj" podStartSLOduration=1.74286312 podStartE2EDuration="6.257417401s" podCreationTimestamp="2026-05-26 11:37:34 +0000 UTC" firstStartedPulling="2026-05-26 11:37:35.423012139 +0000 UTC m=+778.325222450" lastFinishedPulling="2026-05-26 11:37:39.937566423 +0000 UTC m=+782.839776731" observedRunningTime="2026-05-26 11:37:40.256215778 +0000 UTC m=+783.158426109" watchObservedRunningTime="2026-05-26 11:37:40.257417401 +0000 UTC m=+783.159627735"
May 26 11:38:17.624843 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:38:17.624807    2572 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-67566c68b4-9nftj"]
May 26 11:38:17.644281 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:38:17.644257    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-67566c68b4-9nftj"]
May 26 11:38:17.644434 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:38:17.644388    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-67566c68b4-9nftj"
May 26 11:38:17.646969 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:38:17.646944    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
May 26 11:38:17.648809 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:38:17.648783    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-67566c68b4-9nftj"]
May 26 11:38:17.705072 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:38:17.705042    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xq2zr\" (UniqueName: \"kubernetes.io/projected/d2e8a644-a9a2-4cf1-bb98-9a70eb8c44ac-kube-api-access-xq2zr\") pod \"limitador-limitador-67566c68b4-9nftj\" (UID: \"d2e8a644-a9a2-4cf1-bb98-9a70eb8c44ac\") " pod="kuadrant-system/limitador-limitador-67566c68b4-9nftj"
May 26 11:38:17.705251 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:38:17.705077    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/d2e8a644-a9a2-4cf1-bb98-9a70eb8c44ac-config-file\") pod \"limitador-limitador-67566c68b4-9nftj\" (UID: \"d2e8a644-a9a2-4cf1-bb98-9a70eb8c44ac\") " pod="kuadrant-system/limitador-limitador-67566c68b4-9nftj"
May 26 11:38:17.806400 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:38:17.806366    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-xq2zr\" (UniqueName: \"kubernetes.io/projected/d2e8a644-a9a2-4cf1-bb98-9a70eb8c44ac-kube-api-access-xq2zr\") pod \"limitador-limitador-67566c68b4-9nftj\" (UID: \"d2e8a644-a9a2-4cf1-bb98-9a70eb8c44ac\") " pod="kuadrant-system/limitador-limitador-67566c68b4-9nftj"
May 26 11:38:17.806400 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:38:17.806401    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/d2e8a644-a9a2-4cf1-bb98-9a70eb8c44ac-config-file\") pod \"limitador-limitador-67566c68b4-9nftj\" (UID: \"d2e8a644-a9a2-4cf1-bb98-9a70eb8c44ac\") " pod="kuadrant-system/limitador-limitador-67566c68b4-9nftj"
May 26 11:38:17.806959 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:38:17.806941    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/d2e8a644-a9a2-4cf1-bb98-9a70eb8c44ac-config-file\") pod \"limitador-limitador-67566c68b4-9nftj\" (UID: \"d2e8a644-a9a2-4cf1-bb98-9a70eb8c44ac\") " pod="kuadrant-system/limitador-limitador-67566c68b4-9nftj"
May 26 11:38:17.818752 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:38:17.818725    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xq2zr\" (UniqueName: \"kubernetes.io/projected/d2e8a644-a9a2-4cf1-bb98-9a70eb8c44ac-kube-api-access-xq2zr\") pod \"limitador-limitador-67566c68b4-9nftj\" (UID: \"d2e8a644-a9a2-4cf1-bb98-9a70eb8c44ac\") " pod="kuadrant-system/limitador-limitador-67566c68b4-9nftj"
May 26 11:38:17.956244 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:38:17.956209    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-67566c68b4-9nftj"
May 26 11:38:18.088380 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:38:18.088356    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-67566c68b4-9nftj"]
May 26 11:38:18.090448 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:38:18.090422    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2e8a644_a9a2_4cf1_bb98_9a70eb8c44ac.slice/crio-37d8b28ea22704f0794741d613e5f54ce7e43b79b1d540c0a5793723492a9fe5 WatchSource:0}: Error finding container 37d8b28ea22704f0794741d613e5f54ce7e43b79b1d540c0a5793723492a9fe5: Status 404 returned error can't find the container with id 37d8b28ea22704f0794741d613e5f54ce7e43b79b1d540c0a5793723492a9fe5
May 26 11:38:18.357255 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:38:18.357147    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-67566c68b4-9nftj" event={"ID":"d2e8a644-a9a2-4cf1-bb98-9a70eb8c44ac","Type":"ContainerStarted","Data":"37d8b28ea22704f0794741d613e5f54ce7e43b79b1d540c0a5793723492a9fe5"}
May 26 11:38:18.448231 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:38:18.448197    2572 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-79cbc94b89-dvh2b"]
May 26 11:38:18.453200 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:38:18.453172    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-79cbc94b89-dvh2b"
May 26 11:38:18.456454 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:38:18.456434    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-bb9gn\""
May 26 11:38:18.461170 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:38:18.461148    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-79cbc94b89-dvh2b"]
May 26 11:38:18.513290 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:38:18.513264    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4kxd7\" (UniqueName: \"kubernetes.io/projected/d91c59d2-ff2e-4ebc-8c9c-f61d10afe228-kube-api-access-4kxd7\") pod \"authorino-79cbc94b89-dvh2b\" (UID: \"d91c59d2-ff2e-4ebc-8c9c-f61d10afe228\") " pod="kuadrant-system/authorino-79cbc94b89-dvh2b"
May 26 11:38:18.614040 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:38:18.613962    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-4kxd7\" (UniqueName: \"kubernetes.io/projected/d91c59d2-ff2e-4ebc-8c9c-f61d10afe228-kube-api-access-4kxd7\") pod \"authorino-79cbc94b89-dvh2b\" (UID: \"d91c59d2-ff2e-4ebc-8c9c-f61d10afe228\") " pod="kuadrant-system/authorino-79cbc94b89-dvh2b"
May 26 11:38:18.624332 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:38:18.624305    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4kxd7\" (UniqueName: \"kubernetes.io/projected/d91c59d2-ff2e-4ebc-8c9c-f61d10afe228-kube-api-access-4kxd7\") pod \"authorino-79cbc94b89-dvh2b\" (UID: \"d91c59d2-ff2e-4ebc-8c9c-f61d10afe228\") " pod="kuadrant-system/authorino-79cbc94b89-dvh2b"
May 26 11:38:18.764221 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:38:18.764149    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-79cbc94b89-dvh2b"
May 26 11:38:18.889736 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:38:18.889554    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-79cbc94b89-dvh2b"]
May 26 11:38:18.891994 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:38:18.891970    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd91c59d2_ff2e_4ebc_8c9c_f61d10afe228.slice/crio-309a2a2c2d1bf6495f02ae1ef01410534a081227a6aaa7978cea813a1562a783 WatchSource:0}: Error finding container 309a2a2c2d1bf6495f02ae1ef01410534a081227a6aaa7978cea813a1562a783: Status 404 returned error can't find the container with id 309a2a2c2d1bf6495f02ae1ef01410534a081227a6aaa7978cea813a1562a783
May 26 11:38:19.361084 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:38:19.361048    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-79cbc94b89-dvh2b" event={"ID":"d91c59d2-ff2e-4ebc-8c9c-f61d10afe228","Type":"ContainerStarted","Data":"309a2a2c2d1bf6495f02ae1ef01410534a081227a6aaa7978cea813a1562a783"}
May 26 11:38:20.366367 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:38:20.366331    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-67566c68b4-9nftj" event={"ID":"d2e8a644-a9a2-4cf1-bb98-9a70eb8c44ac","Type":"ContainerStarted","Data":"ee2114c8f568d7eee082261570be88ddbec67c684ba77ba17d22931ce94e5b05"}
May 26 11:38:20.366873 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:38:20.366450    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-67566c68b4-9nftj"
May 26 11:38:20.391808 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:38:20.391755    2572 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-67566c68b4-9nftj" podStartSLOduration=1.804217468 podStartE2EDuration="3.391739674s" podCreationTimestamp="2026-05-26 11:38:17 +0000 UTC" firstStartedPulling="2026-05-26 11:38:18.092088721 +0000 UTC m=+820.994299029" lastFinishedPulling="2026-05-26 11:38:19.679610918 +0000 UTC m=+822.581821235" observedRunningTime="2026-05-26 11:38:20.388830086 +0000 UTC m=+823.291040417" watchObservedRunningTime="2026-05-26 11:38:20.391739674 +0000 UTC m=+823.293950003"
May 26 11:38:22.376652 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:38:22.376613    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-79cbc94b89-dvh2b" event={"ID":"d91c59d2-ff2e-4ebc-8c9c-f61d10afe228","Type":"ContainerStarted","Data":"d6041d5af6af962c8c1299de887efbb509d403eaa92d4219cec987c7b6a2e424"}
May 26 11:38:22.396102 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:38:22.396058    2572 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kuadrant-system/authorino-79cbc94b89-dvh2b" podStartSLOduration=1.6193895280000001 podStartE2EDuration="4.396044071s" podCreationTimestamp="2026-05-26 11:38:18 +0000 UTC" firstStartedPulling="2026-05-26 11:38:18.893401608 +0000 UTC m=+821.795611920" lastFinishedPulling="2026-05-26 11:38:21.670056152 +0000 UTC m=+824.572266463" observedRunningTime="2026-05-26 11:38:22.393518156 +0000 UTC m=+825.295728484" watchObservedRunningTime="2026-05-26 11:38:22.396044071 +0000 UTC m=+825.298254399"
May 26 11:38:31.373740 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:38:31.373708    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-67566c68b4-9nftj"
May 26 11:38:42.072517 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:38:42.072480    2572 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-79cbc94b89-dvh2b"]
May 26 11:38:42.072883 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:38:42.072686    2572 kuberuntime_container.go:864] "Killing container with a grace period" pod="kuadrant-system/authorino-79cbc94b89-dvh2b" podUID="d91c59d2-ff2e-4ebc-8c9c-f61d10afe228" containerName="authorino" containerID="cri-o://d6041d5af6af962c8c1299de887efbb509d403eaa92d4219cec987c7b6a2e424" gracePeriod=30
May 26 11:38:42.323615 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:38:42.323558    2572 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-79cbc94b89-dvh2b"
May 26 11:38:42.426900 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:38:42.426869    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4kxd7\" (UniqueName: \"kubernetes.io/projected/d91c59d2-ff2e-4ebc-8c9c-f61d10afe228-kube-api-access-4kxd7\") pod \"d91c59d2-ff2e-4ebc-8c9c-f61d10afe228\" (UID: \"d91c59d2-ff2e-4ebc-8c9c-f61d10afe228\") "
May 26 11:38:42.428994 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:38:42.428966    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d91c59d2-ff2e-4ebc-8c9c-f61d10afe228-kube-api-access-4kxd7" (OuterVolumeSpecName: "kube-api-access-4kxd7") pod "d91c59d2-ff2e-4ebc-8c9c-f61d10afe228" (UID: "d91c59d2-ff2e-4ebc-8c9c-f61d10afe228"). InnerVolumeSpecName "kube-api-access-4kxd7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 11:38:42.447137 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:38:42.447108    2572 generic.go:358] "Generic (PLEG): container finished" podID="d91c59d2-ff2e-4ebc-8c9c-f61d10afe228" containerID="d6041d5af6af962c8c1299de887efbb509d403eaa92d4219cec987c7b6a2e424" exitCode=0
May 26 11:38:42.447257 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:38:42.447157    2572 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-79cbc94b89-dvh2b"
May 26 11:38:42.447257 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:38:42.447181    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-79cbc94b89-dvh2b" event={"ID":"d91c59d2-ff2e-4ebc-8c9c-f61d10afe228","Type":"ContainerDied","Data":"d6041d5af6af962c8c1299de887efbb509d403eaa92d4219cec987c7b6a2e424"}
May 26 11:38:42.447257 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:38:42.447245    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-79cbc94b89-dvh2b" event={"ID":"d91c59d2-ff2e-4ebc-8c9c-f61d10afe228","Type":"ContainerDied","Data":"309a2a2c2d1bf6495f02ae1ef01410534a081227a6aaa7978cea813a1562a783"}
May 26 11:38:42.447397 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:38:42.447269    2572 scope.go:117] "RemoveContainer" containerID="d6041d5af6af962c8c1299de887efbb509d403eaa92d4219cec987c7b6a2e424"
May 26 11:38:42.455432 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:38:42.455416    2572 scope.go:117] "RemoveContainer" containerID="d6041d5af6af962c8c1299de887efbb509d403eaa92d4219cec987c7b6a2e424"
May 26 11:38:42.455684 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:38:42.455666    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6041d5af6af962c8c1299de887efbb509d403eaa92d4219cec987c7b6a2e424\": container with ID starting with d6041d5af6af962c8c1299de887efbb509d403eaa92d4219cec987c7b6a2e424 not found: ID does not exist" containerID="d6041d5af6af962c8c1299de887efbb509d403eaa92d4219cec987c7b6a2e424"
May 26 11:38:42.455746 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:38:42.455695    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6041d5af6af962c8c1299de887efbb509d403eaa92d4219cec987c7b6a2e424"} err="failed to get container status \"d6041d5af6af962c8c1299de887efbb509d403eaa92d4219cec987c7b6a2e424\": rpc error: code = NotFound desc = could not find container \"d6041d5af6af962c8c1299de887efbb509d403eaa92d4219cec987c7b6a2e424\": container with ID starting with d6041d5af6af962c8c1299de887efbb509d403eaa92d4219cec987c7b6a2e424 not found: ID does not exist"
May 26 11:38:42.467216 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:38:42.467149    2572 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-79cbc94b89-dvh2b"]
May 26 11:38:42.468969 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:38:42.468949    2572 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-79cbc94b89-dvh2b"]
May 26 11:38:42.527651 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:38:42.527633    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4kxd7\" (UniqueName: \"kubernetes.io/projected/d91c59d2-ff2e-4ebc-8c9c-f61d10afe228-kube-api-access-4kxd7\") on node \"ip-10-0-136-194.ec2.internal\" DevicePath \"\""
May 26 11:38:43.676210 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:38:43.676152    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d91c59d2-ff2e-4ebc-8c9c-f61d10afe228" path="/var/lib/kubelet/pods/d91c59d2-ff2e-4ebc-8c9c-f61d10afe228/volumes"
May 26 11:39:37.609687 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:39:37.609609    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-5bfgg_c6373342-dd07-4497-a4f3-978ea9613b6e/console-operator/2.log"
May 26 11:39:37.612542 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:39:37.612507    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-5bfgg_c6373342-dd07-4497-a4f3-978ea9613b6e/console-operator/2.log"
May 26 11:39:37.617133 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:39:37.617112    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-js8cc_7264c907-f536-4719-905a-2bbda9e3c336/ovn-acl-logging/0.log"
May 26 11:39:37.619384 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:39:37.619361    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-js8cc_7264c907-f536-4719-905a-2bbda9e3c336/ovn-acl-logging/0.log"
May 26 11:40:29.148102 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:40:29.148066    2572 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve/s3-init-w2bpb"]
May 26 11:40:29.148564 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:40:29.148485    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d91c59d2-ff2e-4ebc-8c9c-f61d10afe228" containerName="authorino"
May 26 11:40:29.148564 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:40:29.148499    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="d91c59d2-ff2e-4ebc-8c9c-f61d10afe228" containerName="authorino"
May 26 11:40:29.148637 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:40:29.148568    2572 memory_manager.go:356] "RemoveStaleState removing state" podUID="d91c59d2-ff2e-4ebc-8c9c-f61d10afe228" containerName="authorino"
May 26 11:40:29.151444 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:40:29.151425    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-w2bpb"
May 26 11:40:29.154238 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:40:29.154216    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
May 26 11:40:29.154351 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:40:29.154251    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
May 26 11:40:29.154415 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:40:29.154354    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-jbblx\""
May 26 11:40:29.154471 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:40:29.154422    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
May 26 11:40:29.154678 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:40:29.154658    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrm26\" (UniqueName: \"kubernetes.io/projected/9acfe7ab-34a1-4bde-9f42-5fa4feed19b8-kube-api-access-wrm26\") pod \"s3-init-w2bpb\" (UID: \"9acfe7ab-34a1-4bde-9f42-5fa4feed19b8\") " pod="kserve/s3-init-w2bpb"
May 26 11:40:29.159070 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:40:29.159049    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-w2bpb"]
May 26 11:40:29.255878 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:40:29.255846    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-wrm26\" (UniqueName: \"kubernetes.io/projected/9acfe7ab-34a1-4bde-9f42-5fa4feed19b8-kube-api-access-wrm26\") pod \"s3-init-w2bpb\" (UID: \"9acfe7ab-34a1-4bde-9f42-5fa4feed19b8\") " pod="kserve/s3-init-w2bpb"
May 26 11:40:29.265253 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:40:29.265221    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrm26\" (UniqueName: \"kubernetes.io/projected/9acfe7ab-34a1-4bde-9f42-5fa4feed19b8-kube-api-access-wrm26\") pod \"s3-init-w2bpb\" (UID: \"9acfe7ab-34a1-4bde-9f42-5fa4feed19b8\") " pod="kserve/s3-init-w2bpb"
May 26 11:40:29.461481 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:40:29.461447    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-w2bpb"
May 26 11:40:29.584984 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:40:29.584959    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-w2bpb"]
May 26 11:40:29.587355 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:40:29.587324    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9acfe7ab_34a1_4bde_9f42_5fa4feed19b8.slice/crio-d7cc84a5c677e6e3dc398a250613867ead8af863cf2cd436a0cceb623364ccfe WatchSource:0}: Error finding container d7cc84a5c677e6e3dc398a250613867ead8af863cf2cd436a0cceb623364ccfe: Status 404 returned error can't find the container with id d7cc84a5c677e6e3dc398a250613867ead8af863cf2cd436a0cceb623364ccfe
May 26 11:40:29.589072 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:40:29.589053    2572 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 26 11:40:29.811597 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:40:29.811515    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-w2bpb" event={"ID":"9acfe7ab-34a1-4bde-9f42-5fa4feed19b8","Type":"ContainerStarted","Data":"d7cc84a5c677e6e3dc398a250613867ead8af863cf2cd436a0cceb623364ccfe"}
May 26 11:40:34.832696 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:40:34.832657    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-w2bpb" event={"ID":"9acfe7ab-34a1-4bde-9f42-5fa4feed19b8","Type":"ContainerStarted","Data":"071a05ca32205eb01ed355f53421e9ce5033be5defd47a649fe3ed16dc87cba1"}
May 26 11:40:34.847936 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:40:34.847878    2572 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve/s3-init-w2bpb" podStartSLOduration=1.439863576 podStartE2EDuration="5.847860183s" podCreationTimestamp="2026-05-26 11:40:29 +0000 UTC" firstStartedPulling="2026-05-26 11:40:29.589179664 +0000 UTC m=+952.491389971" lastFinishedPulling="2026-05-26 11:40:33.997176268 +0000 UTC m=+956.899386578" observedRunningTime="2026-05-26 11:40:34.847257662 +0000 UTC m=+957.749467990" watchObservedRunningTime="2026-05-26 11:40:34.847860183 +0000 UTC m=+957.750070514"
May 26 11:40:37.844555 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:40:37.844520    2572 generic.go:358] "Generic (PLEG): container finished" podID="9acfe7ab-34a1-4bde-9f42-5fa4feed19b8" containerID="071a05ca32205eb01ed355f53421e9ce5033be5defd47a649fe3ed16dc87cba1" exitCode=0
May 26 11:40:37.844936 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:40:37.844593    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-w2bpb" event={"ID":"9acfe7ab-34a1-4bde-9f42-5fa4feed19b8","Type":"ContainerDied","Data":"071a05ca32205eb01ed355f53421e9ce5033be5defd47a649fe3ed16dc87cba1"}
May 26 11:40:38.980258 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:40:38.980232    2572 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-w2bpb"
May 26 11:40:39.045312 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:40:39.045281    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wrm26\" (UniqueName: \"kubernetes.io/projected/9acfe7ab-34a1-4bde-9f42-5fa4feed19b8-kube-api-access-wrm26\") pod \"9acfe7ab-34a1-4bde-9f42-5fa4feed19b8\" (UID: \"9acfe7ab-34a1-4bde-9f42-5fa4feed19b8\") "
May 26 11:40:39.047355 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:40:39.047327    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9acfe7ab-34a1-4bde-9f42-5fa4feed19b8-kube-api-access-wrm26" (OuterVolumeSpecName: "kube-api-access-wrm26") pod "9acfe7ab-34a1-4bde-9f42-5fa4feed19b8" (UID: "9acfe7ab-34a1-4bde-9f42-5fa4feed19b8"). InnerVolumeSpecName "kube-api-access-wrm26". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 11:40:39.146248 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:40:39.146161    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wrm26\" (UniqueName: \"kubernetes.io/projected/9acfe7ab-34a1-4bde-9f42-5fa4feed19b8-kube-api-access-wrm26\") on node \"ip-10-0-136-194.ec2.internal\" DevicePath \"\""
May 26 11:40:39.852947 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:40:39.852913    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-w2bpb" event={"ID":"9acfe7ab-34a1-4bde-9f42-5fa4feed19b8","Type":"ContainerDied","Data":"d7cc84a5c677e6e3dc398a250613867ead8af863cf2cd436a0cceb623364ccfe"}
May 26 11:40:39.852947 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:40:39.852949    2572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d7cc84a5c677e6e3dc398a250613867ead8af863cf2cd436a0cceb623364ccfe"
May 26 11:40:39.852947 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:40:39.852930    2572 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-w2bpb"
May 26 11:41:21.422886 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:41:21.422852    2572 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-754cf65554-plnxv"]
May 26 11:41:21.423409 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:41:21.423252    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9acfe7ab-34a1-4bde-9f42-5fa4feed19b8" containerName="s3-init"
May 26 11:41:21.423409 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:41:21.423265    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="9acfe7ab-34a1-4bde-9f42-5fa4feed19b8" containerName="s3-init"
May 26 11:41:21.423409 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:41:21.423340    2572 memory_manager.go:356] "RemoveStaleState removing state" podUID="9acfe7ab-34a1-4bde-9f42-5fa4feed19b8" containerName="s3-init"
May 26 11:41:21.426401 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:41:21.426382    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-754cf65554-plnxv"
May 26 11:41:21.429164 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:41:21.429144    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
May 26 11:41:21.430244 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:41:21.430226    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-qjhlk\""
May 26 11:41:21.430244 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:41:21.430227    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-inline-config-test-kserve-self-signed-certs\""
May 26 11:41:21.430361 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:41:21.430182    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
May 26 11:41:21.437252 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:41:21.437231    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-754cf65554-plnxv"]
May 26 11:41:21.600861 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:41:21.600819    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/765046db-e719-4f14-9bbb-15ab16f5c946-dshm\") pod \"scheduler-inline-config-test-kserve-754cf65554-plnxv\" (UID: \"765046db-e719-4f14-9bbb-15ab16f5c946\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-754cf65554-plnxv"
May 26 11:41:21.600861 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:41:21.600860    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6rfb\" (UniqueName: \"kubernetes.io/projected/765046db-e719-4f14-9bbb-15ab16f5c946-kube-api-access-h6rfb\") pod \"scheduler-inline-config-test-kserve-754cf65554-plnxv\" (UID: \"765046db-e719-4f14-9bbb-15ab16f5c946\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-754cf65554-plnxv"
May 26 11:41:21.601089 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:41:21.600898    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/765046db-e719-4f14-9bbb-15ab16f5c946-tls-certs\") pod \"scheduler-inline-config-test-kserve-754cf65554-plnxv\" (UID: \"765046db-e719-4f14-9bbb-15ab16f5c946\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-754cf65554-plnxv"
May 26 11:41:21.601089 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:41:21.600928    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/765046db-e719-4f14-9bbb-15ab16f5c946-tmp-dir\") pod \"scheduler-inline-config-test-kserve-754cf65554-plnxv\" (UID: \"765046db-e719-4f14-9bbb-15ab16f5c946\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-754cf65554-plnxv"
May 26 11:41:21.601089 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:41:21.601061    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/765046db-e719-4f14-9bbb-15ab16f5c946-home\") pod \"scheduler-inline-config-test-kserve-754cf65554-plnxv\" (UID: \"765046db-e719-4f14-9bbb-15ab16f5c946\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-754cf65554-plnxv"
May 26 11:41:21.601258 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:41:21.601095    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/765046db-e719-4f14-9bbb-15ab16f5c946-model-cache\") pod \"scheduler-inline-config-test-kserve-754cf65554-plnxv\" (UID: \"765046db-e719-4f14-9bbb-15ab16f5c946\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-754cf65554-plnxv"
May 26 11:41:21.601258 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:41:21.601112    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/765046db-e719-4f14-9bbb-15ab16f5c946-kserve-provision-location\") pod \"scheduler-inline-config-test-kserve-754cf65554-plnxv\" (UID: \"765046db-e719-4f14-9bbb-15ab16f5c946\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-754cf65554-plnxv"
May 26 11:41:21.701873 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:41:21.701778    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/765046db-e719-4f14-9bbb-15ab16f5c946-dshm\") pod \"scheduler-inline-config-test-kserve-754cf65554-plnxv\" (UID: \"765046db-e719-4f14-9bbb-15ab16f5c946\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-754cf65554-plnxv"
May 26 11:41:21.701873 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:41:21.701817    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-h6rfb\" (UniqueName: \"kubernetes.io/projected/765046db-e719-4f14-9bbb-15ab16f5c946-kube-api-access-h6rfb\") pod \"scheduler-inline-config-test-kserve-754cf65554-plnxv\" (UID: \"765046db-e719-4f14-9bbb-15ab16f5c946\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-754cf65554-plnxv"
May 26 11:41:21.701873 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:41:21.701853    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/765046db-e719-4f14-9bbb-15ab16f5c946-tls-certs\") pod \"scheduler-inline-config-test-kserve-754cf65554-plnxv\" (UID: \"765046db-e719-4f14-9bbb-15ab16f5c946\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-754cf65554-plnxv"
May 26 11:41:21.701873 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:41:21.701874    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/765046db-e719-4f14-9bbb-15ab16f5c946-tmp-dir\") pod \"scheduler-inline-config-test-kserve-754cf65554-plnxv\" (UID: \"765046db-e719-4f14-9bbb-15ab16f5c946\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-754cf65554-plnxv"
May 26 11:41:21.702228 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:41:21.701942    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/765046db-e719-4f14-9bbb-15ab16f5c946-home\") pod \"scheduler-inline-config-test-kserve-754cf65554-plnxv\" (UID: \"765046db-e719-4f14-9bbb-15ab16f5c946\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-754cf65554-plnxv"
May 26 11:41:21.702228 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:41:21.701966    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/765046db-e719-4f14-9bbb-15ab16f5c946-model-cache\") pod \"scheduler-inline-config-test-kserve-754cf65554-plnxv\" (UID: \"765046db-e719-4f14-9bbb-15ab16f5c946\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-754cf65554-plnxv"
May 26 11:41:21.702228 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:41:21.701991    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/765046db-e719-4f14-9bbb-15ab16f5c946-kserve-provision-location\") pod \"scheduler-inline-config-test-kserve-754cf65554-plnxv\" (UID: \"765046db-e719-4f14-9bbb-15ab16f5c946\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-754cf65554-plnxv"
May 26 11:41:21.702401 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:41:21.702291    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/765046db-e719-4f14-9bbb-15ab16f5c946-tmp-dir\") pod \"scheduler-inline-config-test-kserve-754cf65554-plnxv\" (UID: \"765046db-e719-4f14-9bbb-15ab16f5c946\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-754cf65554-plnxv"
May 26 11:41:21.702453 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:41:21.702403    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/765046db-e719-4f14-9bbb-15ab16f5c946-kserve-provision-location\") pod \"scheduler-inline-config-test-kserve-754cf65554-plnxv\" (UID: \"765046db-e719-4f14-9bbb-15ab16f5c946\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-754cf65554-plnxv"
May 26 11:41:21.702453 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:41:21.702424    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/765046db-e719-4f14-9bbb-15ab16f5c946-home\") pod \"scheduler-inline-config-test-kserve-754cf65554-plnxv\" (UID: \"765046db-e719-4f14-9bbb-15ab16f5c946\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-754cf65554-plnxv"
May 26 11:41:21.702617 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:41:21.702463    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/765046db-e719-4f14-9bbb-15ab16f5c946-model-cache\") pod \"scheduler-inline-config-test-kserve-754cf65554-plnxv\" (UID: \"765046db-e719-4f14-9bbb-15ab16f5c946\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-754cf65554-plnxv"
May 26 11:41:21.704163 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:41:21.704143    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/765046db-e719-4f14-9bbb-15ab16f5c946-dshm\") pod \"scheduler-inline-config-test-kserve-754cf65554-plnxv\" (UID: \"765046db-e719-4f14-9bbb-15ab16f5c946\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-754cf65554-plnxv"
May 26 11:41:21.704474 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:41:21.704457    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/765046db-e719-4f14-9bbb-15ab16f5c946-tls-certs\") pod \"scheduler-inline-config-test-kserve-754cf65554-plnxv\" (UID: \"765046db-e719-4f14-9bbb-15ab16f5c946\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-754cf65554-plnxv"
May 26 11:41:21.711490 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:41:21.711470    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6rfb\" (UniqueName: \"kubernetes.io/projected/765046db-e719-4f14-9bbb-15ab16f5c946-kube-api-access-h6rfb\") pod \"scheduler-inline-config-test-kserve-754cf65554-plnxv\" (UID: \"765046db-e719-4f14-9bbb-15ab16f5c946\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-754cf65554-plnxv"
May 26 11:41:21.737225 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:41:21.737202    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-754cf65554-plnxv"
May 26 11:41:21.868152 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:41:21.868047    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-754cf65554-plnxv"]
May 26 11:41:21.870707 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:41:21.870679    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod765046db_e719_4f14_9bbb_15ab16f5c946.slice/crio-a840b7f9e25cec61756b3d799249ccd45565015cf180a6745179c3067ff5300c WatchSource:0}: Error finding container a840b7f9e25cec61756b3d799249ccd45565015cf180a6745179c3067ff5300c: Status 404 returned error can't find the container with id a840b7f9e25cec61756b3d799249ccd45565015cf180a6745179c3067ff5300c
May 26 11:41:21.991318 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:41:21.991236    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-754cf65554-plnxv" event={"ID":"765046db-e719-4f14-9bbb-15ab16f5c946","Type":"ContainerStarted","Data":"a840b7f9e25cec61756b3d799249ccd45565015cf180a6745179c3067ff5300c"}
May 26 11:41:26.010621 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:41:26.010586    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-754cf65554-plnxv" event={"ID":"765046db-e719-4f14-9bbb-15ab16f5c946","Type":"ContainerStarted","Data":"d3acc31fbb8ee1e975209f5d2d299afc29eef49861d995ec4996afddd4d47ea0"}
May 26 11:42:03.827921 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:42:03.827888    2572 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-74545d684b69k95"]
May 26 11:42:03.830388 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:42:03.830371    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-74545d684b69k95"
May 26 11:42:03.833067 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:42:03.833040    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvad71fa5348b85aebd404221bba611457-kserve-self-signed-certs\""
May 26 11:42:03.841823 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:42:03.841795    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-74545d684b69k95"]
May 26 11:42:03.979288 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:42:03.979256    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c43aaf2-a973-4b5e-8eba-ad97c89d7555-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-74545d684b69k95\" (UID: \"2c43aaf2-a973-4b5e-8eba-ad97c89d7555\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-74545d684b69k95"
May 26 11:42:03.979288 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:42:03.979287    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c43aaf2-a973-4b5e-8eba-ad97c89d7555-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-74545d684b69k95\" (UID: \"2c43aaf2-a973-4b5e-8eba-ad97c89d7555\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-74545d684b69k95"
May 26 11:42:03.979505 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:42:03.979373    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c43aaf2-a973-4b5e-8eba-ad97c89d7555-home\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-74545d684b69k95\" (UID: \"2c43aaf2-a973-4b5e-8eba-ad97c89d7555\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-74545d684b69k95"
May 26 11:42:03.979505 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:42:03.979403    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmhpk\" (UniqueName: \"kubernetes.io/projected/2c43aaf2-a973-4b5e-8eba-ad97c89d7555-kube-api-access-xmhpk\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-74545d684b69k95\" (UID: \"2c43aaf2-a973-4b5e-8eba-ad97c89d7555\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-74545d684b69k95"
May 26 11:42:03.979505 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:42:03.979427    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c43aaf2-a973-4b5e-8eba-ad97c89d7555-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-74545d684b69k95\" (UID: \"2c43aaf2-a973-4b5e-8eba-ad97c89d7555\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-74545d684b69k95"
May 26 11:42:03.979505 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:42:03.979463    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c43aaf2-a973-4b5e-8eba-ad97c89d7555-dshm\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-74545d684b69k95\" (UID: \"2c43aaf2-a973-4b5e-8eba-ad97c89d7555\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-74545d684b69k95"
May 26 11:42:03.979652 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:42:03.979504    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2c43aaf2-a973-4b5e-8eba-ad97c89d7555-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-74545d684b69k95\" (UID: \"2c43aaf2-a973-4b5e-8eba-ad97c89d7555\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-74545d684b69k95"
May 26 11:42:04.080723 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:42:04.080641    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c43aaf2-a973-4b5e-8eba-ad97c89d7555-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-74545d684b69k95\" (UID: \"2c43aaf2-a973-4b5e-8eba-ad97c89d7555\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-74545d684b69k95"
May 26 11:42:04.080723 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:42:04.080686    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c43aaf2-a973-4b5e-8eba-ad97c89d7555-dshm\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-74545d684b69k95\" (UID: \"2c43aaf2-a973-4b5e-8eba-ad97c89d7555\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-74545d684b69k95"
May 26 11:42:04.080723 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:42:04.080709    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2c43aaf2-a973-4b5e-8eba-ad97c89d7555-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-74545d684b69k95\" (UID: \"2c43aaf2-a973-4b5e-8eba-ad97c89d7555\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-74545d684b69k95"
May 26 11:42:04.080962 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:42:04.080793    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c43aaf2-a973-4b5e-8eba-ad97c89d7555-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-74545d684b69k95\" (UID: \"2c43aaf2-a973-4b5e-8eba-ad97c89d7555\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-74545d684b69k95"
May 26 11:42:04.080962 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:42:04.080820    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c43aaf2-a973-4b5e-8eba-ad97c89d7555-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-74545d684b69k95\" (UID: \"2c43aaf2-a973-4b5e-8eba-ad97c89d7555\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-74545d684b69k95"
May 26 11:42:04.080962 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:42:04.080872    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c43aaf2-a973-4b5e-8eba-ad97c89d7555-home\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-74545d684b69k95\" (UID: \"2c43aaf2-a973-4b5e-8eba-ad97c89d7555\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-74545d684b69k95"
May 26 11:42:04.080962 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:42:04.080896    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-xmhpk\" (UniqueName: \"kubernetes.io/projected/2c43aaf2-a973-4b5e-8eba-ad97c89d7555-kube-api-access-xmhpk\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-74545d684b69k95\" (UID: \"2c43aaf2-a973-4b5e-8eba-ad97c89d7555\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-74545d684b69k95"
May 26 11:42:04.081161 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:42:04.081069    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c43aaf2-a973-4b5e-8eba-ad97c89d7555-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-74545d684b69k95\" (UID: \"2c43aaf2-a973-4b5e-8eba-ad97c89d7555\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-74545d684b69k95"
May 26 11:42:04.081320 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:42:04.081295    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c43aaf2-a973-4b5e-8eba-ad97c89d7555-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-74545d684b69k95\" (UID: \"2c43aaf2-a973-4b5e-8eba-ad97c89d7555\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-74545d684b69k95"
May 26 11:42:04.081412 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:42:04.081327    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c43aaf2-a973-4b5e-8eba-ad97c89d7555-home\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-74545d684b69k95\" (UID: \"2c43aaf2-a973-4b5e-8eba-ad97c89d7555\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-74545d684b69k95"
May 26 11:42:04.081455 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:42:04.081425    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2c43aaf2-a973-4b5e-8eba-ad97c89d7555-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-74545d684b69k95\" (UID: \"2c43aaf2-a973-4b5e-8eba-ad97c89d7555\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-74545d684b69k95"
May 26 11:42:04.083040 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:42:04.083023    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c43aaf2-a973-4b5e-8eba-ad97c89d7555-dshm\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-74545d684b69k95\" (UID: \"2c43aaf2-a973-4b5e-8eba-ad97c89d7555\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-74545d684b69k95"
May 26 11:42:04.083298 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:42:04.083281    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c43aaf2-a973-4b5e-8eba-ad97c89d7555-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-74545d684b69k95\" (UID: \"2c43aaf2-a973-4b5e-8eba-ad97c89d7555\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-74545d684b69k95"
May 26 11:42:04.088326 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:42:04.088303    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmhpk\" (UniqueName: \"kubernetes.io/projected/2c43aaf2-a973-4b5e-8eba-ad97c89d7555-kube-api-access-xmhpk\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-74545d684b69k95\" (UID: \"2c43aaf2-a973-4b5e-8eba-ad97c89d7555\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-74545d684b69k95"
May 26 11:42:04.141105 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:42:04.141084    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-74545d684b69k95"
May 26 11:42:04.273924 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:42:04.273900    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-74545d684b69k95"]
May 26 11:42:04.276704 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:42:04.276675    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c43aaf2_a973_4b5e_8eba_ad97c89d7555.slice/crio-c1279f30978e2fd1fd51d58be02072234ee9f00cfd78b93d337e176c19601fe7 WatchSource:0}: Error finding container c1279f30978e2fd1fd51d58be02072234ee9f00cfd78b93d337e176c19601fe7: Status 404 returned error can't find the container with id c1279f30978e2fd1fd51d58be02072234ee9f00cfd78b93d337e176c19601fe7
May 26 11:42:05.149568 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:42:05.149486    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-74545d684b69k95" event={"ID":"2c43aaf2-a973-4b5e-8eba-ad97c89d7555","Type":"ContainerStarted","Data":"150242e9d6b303ff70f009a3912b19edcc81e60b751655651637bac268de065b"}
May 26 11:42:05.149568 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:42:05.149533    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-74545d684b69k95" event={"ID":"2c43aaf2-a973-4b5e-8eba-ad97c89d7555","Type":"ContainerStarted","Data":"c1279f30978e2fd1fd51d58be02072234ee9f00cfd78b93d337e176c19601fe7"}
May 26 11:42:12.176068 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:42:12.176035    2572 generic.go:358] "Generic (PLEG): container finished" podID="2c43aaf2-a973-4b5e-8eba-ad97c89d7555" containerID="150242e9d6b303ff70f009a3912b19edcc81e60b751655651637bac268de065b" exitCode=0
May 26 11:42:12.176486 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:42:12.176109    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-74545d684b69k95" event={"ID":"2c43aaf2-a973-4b5e-8eba-ad97c89d7555","Type":"ContainerDied","Data":"150242e9d6b303ff70f009a3912b19edcc81e60b751655651637bac268de065b"}
May 26 11:42:14.184982 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:42:14.184946    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-74545d684b69k95" event={"ID":"2c43aaf2-a973-4b5e-8eba-ad97c89d7555","Type":"ContainerStarted","Data":"d43279bed073aa1434c3624c05995f6427812d3d95f3f4f7037e76331ba91bbe"}
May 26 11:42:14.206001 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:42:14.205951    2572 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-74545d684b69k95" podStartSLOduration=10.191004834 podStartE2EDuration="11.205936361s" podCreationTimestamp="2026-05-26 11:42:03 +0000 UTC" firstStartedPulling="2026-05-26 11:42:12.177345378 +0000 UTC m=+1055.079555685" lastFinishedPulling="2026-05-26 11:42:13.192276887 +0000 UTC m=+1056.094487212" observedRunningTime="2026-05-26 11:42:14.2045094 +0000 UTC m=+1057.106719729" watchObservedRunningTime="2026-05-26 11:42:14.205936361 +0000 UTC m=+1057.108146693"
May 26 11:42:24.141685 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:42:24.141596    2572 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-74545d684b69k95"
May 26 11:42:24.141685 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:42:24.141652    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-74545d684b69k95"
May 26 11:42:24.154019 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:42:24.153994    2572 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-74545d684b69k95"
May 26 11:42:24.230034 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:42:24.230004    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-74545d684b69k95"
May 26 11:42:24.792018 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:42:24.791984    2572 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-74545d684b69k95"]
May 26 11:42:25.270701 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:42:25.270673    2572 secret.go:189] Couldn't get secret kserve-ci-e2e-test/llmisvad71fa5348b85aebd404221bba611457-kserve-self-signed-certs: secret "llmisvad71fa5348b85aebd404221bba611457-kserve-self-signed-certs" not found
May 26 11:42:25.271082 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:42:25.270754    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c43aaf2-a973-4b5e-8eba-ad97c89d7555-tls-certs podName:2c43aaf2-a973-4b5e-8eba-ad97c89d7555 nodeName:}" failed. No retries permitted until 2026-05-26 11:42:25.770736192 +0000 UTC m=+1068.672946502 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/2c43aaf2-a973-4b5e-8eba-ad97c89d7555-tls-certs") pod "llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-74545d684b69k95" (UID: "2c43aaf2-a973-4b5e-8eba-ad97c89d7555") : secret "llmisvad71fa5348b85aebd404221bba611457-kserve-self-signed-certs" not found
May 26 11:42:25.775303 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:42:25.775268    2572 secret.go:189] Couldn't get secret kserve-ci-e2e-test/llmisvad71fa5348b85aebd404221bba611457-kserve-self-signed-certs: secret "llmisvad71fa5348b85aebd404221bba611457-kserve-self-signed-certs" not found
May 26 11:42:25.775478 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:42:25.775348    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c43aaf2-a973-4b5e-8eba-ad97c89d7555-tls-certs podName:2c43aaf2-a973-4b5e-8eba-ad97c89d7555 nodeName:}" failed. No retries permitted until 2026-05-26 11:42:26.775333019 +0000 UTC m=+1069.677543326 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/2c43aaf2-a973-4b5e-8eba-ad97c89d7555-tls-certs") pod "llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-74545d684b69k95" (UID: "2c43aaf2-a973-4b5e-8eba-ad97c89d7555") : secret "llmisvad71fa5348b85aebd404221bba611457-kserve-self-signed-certs" not found
May 26 11:42:26.225922 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:42:26.225883    2572 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-74545d684b69k95" podUID="2c43aaf2-a973-4b5e-8eba-ad97c89d7555" containerName="main" containerID="cri-o://d43279bed073aa1434c3624c05995f6427812d3d95f3f4f7037e76331ba91bbe" gracePeriod=30
May 26 11:42:26.498431 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:42:26.498409    2572 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-74545d684b69k95"
May 26 11:42:26.583380 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:42:26.583345    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c43aaf2-a973-4b5e-8eba-ad97c89d7555-tls-certs\") pod \"2c43aaf2-a973-4b5e-8eba-ad97c89d7555\" (UID: \"2c43aaf2-a973-4b5e-8eba-ad97c89d7555\") "
May 26 11:42:26.583563 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:42:26.583412    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xmhpk\" (UniqueName: \"kubernetes.io/projected/2c43aaf2-a973-4b5e-8eba-ad97c89d7555-kube-api-access-xmhpk\") pod \"2c43aaf2-a973-4b5e-8eba-ad97c89d7555\" (UID: \"2c43aaf2-a973-4b5e-8eba-ad97c89d7555\") "
May 26 11:42:26.583563 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:42:26.583445    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c43aaf2-a973-4b5e-8eba-ad97c89d7555-tmp-dir\") pod \"2c43aaf2-a973-4b5e-8eba-ad97c89d7555\" (UID: \"2c43aaf2-a973-4b5e-8eba-ad97c89d7555\") "
May 26 11:42:26.583563 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:42:26.583506    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c43aaf2-a973-4b5e-8eba-ad97c89d7555-model-cache\") pod \"2c43aaf2-a973-4b5e-8eba-ad97c89d7555\" (UID: \"2c43aaf2-a973-4b5e-8eba-ad97c89d7555\") "
May 26 11:42:26.583563 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:42:26.583526    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c43aaf2-a973-4b5e-8eba-ad97c89d7555-dshm\") pod \"2c43aaf2-a973-4b5e-8eba-ad97c89d7555\" (UID: \"2c43aaf2-a973-4b5e-8eba-ad97c89d7555\") "
May 26 11:42:26.583822 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:42:26.583578    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2c43aaf2-a973-4b5e-8eba-ad97c89d7555-kserve-provision-location\") pod \"2c43aaf2-a973-4b5e-8eba-ad97c89d7555\" (UID: \"2c43aaf2-a973-4b5e-8eba-ad97c89d7555\") "
May 26 11:42:26.583822 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:42:26.583608    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c43aaf2-a973-4b5e-8eba-ad97c89d7555-home\") pod \"2c43aaf2-a973-4b5e-8eba-ad97c89d7555\" (UID: \"2c43aaf2-a973-4b5e-8eba-ad97c89d7555\") "
May 26 11:42:26.583918 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:42:26.583817    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c43aaf2-a973-4b5e-8eba-ad97c89d7555-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2c43aaf2-a973-4b5e-8eba-ad97c89d7555" (UID: "2c43aaf2-a973-4b5e-8eba-ad97c89d7555"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 11:42:26.583918 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:42:26.583803    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c43aaf2-a973-4b5e-8eba-ad97c89d7555-model-cache" (OuterVolumeSpecName: "model-cache") pod "2c43aaf2-a973-4b5e-8eba-ad97c89d7555" (UID: "2c43aaf2-a973-4b5e-8eba-ad97c89d7555"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 11:42:26.584028 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:42:26.583952    2572 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c43aaf2-a973-4b5e-8eba-ad97c89d7555-model-cache\") on node \"ip-10-0-136-194.ec2.internal\" DevicePath \"\""
May 26 11:42:26.584028 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:42:26.583973    2572 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c43aaf2-a973-4b5e-8eba-ad97c89d7555-tmp-dir\") on node \"ip-10-0-136-194.ec2.internal\" DevicePath \"\""
May 26 11:42:26.584124 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:42:26.584052    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c43aaf2-a973-4b5e-8eba-ad97c89d7555-home" (OuterVolumeSpecName: "home") pod "2c43aaf2-a973-4b5e-8eba-ad97c89d7555" (UID: "2c43aaf2-a973-4b5e-8eba-ad97c89d7555"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 11:42:26.585682 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:42:26.585661    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c43aaf2-a973-4b5e-8eba-ad97c89d7555-kube-api-access-xmhpk" (OuterVolumeSpecName: "kube-api-access-xmhpk") pod "2c43aaf2-a973-4b5e-8eba-ad97c89d7555" (UID: "2c43aaf2-a973-4b5e-8eba-ad97c89d7555"). InnerVolumeSpecName "kube-api-access-xmhpk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 11:42:26.585902 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:42:26.585879    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c43aaf2-a973-4b5e-8eba-ad97c89d7555-dshm" (OuterVolumeSpecName: "dshm") pod "2c43aaf2-a973-4b5e-8eba-ad97c89d7555" (UID: "2c43aaf2-a973-4b5e-8eba-ad97c89d7555"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 11:42:26.585993 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:42:26.585922    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c43aaf2-a973-4b5e-8eba-ad97c89d7555-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2c43aaf2-a973-4b5e-8eba-ad97c89d7555" (UID: "2c43aaf2-a973-4b5e-8eba-ad97c89d7555"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 11:42:26.684679 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:42:26.684648    2572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c43aaf2-a973-4b5e-8eba-ad97c89d7555-tls-certs\") on node \"ip-10-0-136-194.ec2.internal\" DevicePath \"\""
May 26 11:42:26.684679 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:42:26.684676    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xmhpk\" (UniqueName: \"kubernetes.io/projected/2c43aaf2-a973-4b5e-8eba-ad97c89d7555-kube-api-access-xmhpk\") on node \"ip-10-0-136-194.ec2.internal\" DevicePath \"\""
May 26 11:42:26.684876 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:42:26.684691    2572 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c43aaf2-a973-4b5e-8eba-ad97c89d7555-dshm\") on node \"ip-10-0-136-194.ec2.internal\" DevicePath \"\""
May 26 11:42:26.684876 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:42:26.684701    2572 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c43aaf2-a973-4b5e-8eba-ad97c89d7555-home\") on node \"ip-10-0-136-194.ec2.internal\" DevicePath \"\""
May 26 11:42:26.954640 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:42:26.954599    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c43aaf2-a973-4b5e-8eba-ad97c89d7555-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2c43aaf2-a973-4b5e-8eba-ad97c89d7555" (UID: "2c43aaf2-a973-4b5e-8eba-ad97c89d7555"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 11:42:26.987493 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:42:26.987465    2572 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2c43aaf2-a973-4b5e-8eba-ad97c89d7555-kserve-provision-location\") on node \"ip-10-0-136-194.ec2.internal\" DevicePath \"\""
May 26 11:42:27.230527 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:42:27.230442    2572 generic.go:358] "Generic (PLEG): container finished" podID="2c43aaf2-a973-4b5e-8eba-ad97c89d7555" containerID="d43279bed073aa1434c3624c05995f6427812d3d95f3f4f7037e76331ba91bbe" exitCode=0
May 26 11:42:27.230527 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:42:27.230514    2572 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-74545d684b69k95"
May 26 11:42:27.230754 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:42:27.230531    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-74545d684b69k95" event={"ID":"2c43aaf2-a973-4b5e-8eba-ad97c89d7555","Type":"ContainerDied","Data":"d43279bed073aa1434c3624c05995f6427812d3d95f3f4f7037e76331ba91bbe"}
May 26 11:42:27.230754 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:42:27.230576    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-74545d684b69k95" event={"ID":"2c43aaf2-a973-4b5e-8eba-ad97c89d7555","Type":"ContainerDied","Data":"c1279f30978e2fd1fd51d58be02072234ee9f00cfd78b93d337e176c19601fe7"}
May 26 11:42:27.230754 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:42:27.230598    2572 scope.go:117] "RemoveContainer" containerID="d43279bed073aa1434c3624c05995f6427812d3d95f3f4f7037e76331ba91bbe"
May 26 11:42:27.239643 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:42:27.239628    2572 scope.go:117] "RemoveContainer" containerID="150242e9d6b303ff70f009a3912b19edcc81e60b751655651637bac268de065b"
May 26 11:42:27.252743 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:42:27.252709    2572 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-74545d684b69k95"]
May 26 11:42:27.256269 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:42:27.256244    2572 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-74545d684b69k95"]
May 26 11:42:27.302859 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:42:27.302835    2572 scope.go:117] "RemoveContainer" containerID="d43279bed073aa1434c3624c05995f6427812d3d95f3f4f7037e76331ba91bbe"
May 26 11:42:27.303235 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:42:27.303178    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d43279bed073aa1434c3624c05995f6427812d3d95f3f4f7037e76331ba91bbe\": container with ID starting with d43279bed073aa1434c3624c05995f6427812d3d95f3f4f7037e76331ba91bbe not found: ID does not exist" containerID="d43279bed073aa1434c3624c05995f6427812d3d95f3f4f7037e76331ba91bbe"
May 26 11:42:27.303290 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:42:27.303242    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d43279bed073aa1434c3624c05995f6427812d3d95f3f4f7037e76331ba91bbe"} err="failed to get container status \"d43279bed073aa1434c3624c05995f6427812d3d95f3f4f7037e76331ba91bbe\": rpc error: code = NotFound desc = could not find container \"d43279bed073aa1434c3624c05995f6427812d3d95f3f4f7037e76331ba91bbe\": container with ID starting with d43279bed073aa1434c3624c05995f6427812d3d95f3f4f7037e76331ba91bbe not found: ID does not exist"
May 26 11:42:27.303290 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:42:27.303271    2572 scope.go:117] "RemoveContainer" containerID="150242e9d6b303ff70f009a3912b19edcc81e60b751655651637bac268de065b"
May 26 11:42:27.303546 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:42:27.303527    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"150242e9d6b303ff70f009a3912b19edcc81e60b751655651637bac268de065b\": container with ID starting with 150242e9d6b303ff70f009a3912b19edcc81e60b751655651637bac268de065b not found: ID does not exist" containerID="150242e9d6b303ff70f009a3912b19edcc81e60b751655651637bac268de065b"
May 26 11:42:27.303595 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:42:27.303551    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"150242e9d6b303ff70f009a3912b19edcc81e60b751655651637bac268de065b"} err="failed to get container status \"150242e9d6b303ff70f009a3912b19edcc81e60b751655651637bac268de065b\": rpc error: code = NotFound desc = could not find container \"150242e9d6b303ff70f009a3912b19edcc81e60b751655651637bac268de065b\": container with ID starting with 150242e9d6b303ff70f009a3912b19edcc81e60b751655651637bac268de065b not found: ID does not exist"
May 26 11:42:27.676857 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:42:27.676823    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2c43aaf2-a973-4b5e-8eba-ad97c89d7555" path="/var/lib/kubelet/pods/2c43aaf2-a973-4b5e-8eba-ad97c89d7555/volumes"
May 26 11:42:42.235529 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:42:42.235490    2572 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-b9b644d5d-54xv9"]
May 26 11:42:42.235907 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:42:42.235894    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c43aaf2-a973-4b5e-8eba-ad97c89d7555" containerName="main"
May 26 11:42:42.235950 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:42:42.235908    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c43aaf2-a973-4b5e-8eba-ad97c89d7555" containerName="main"
May 26 11:42:42.235950 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:42:42.235921    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c43aaf2-a973-4b5e-8eba-ad97c89d7555" containerName="storage-initializer"
May 26 11:42:42.235950 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:42:42.235927    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c43aaf2-a973-4b5e-8eba-ad97c89d7555" containerName="storage-initializer"
May 26 11:42:42.236039 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:42:42.235987    2572 memory_manager.go:356] "RemoveStaleState removing state" podUID="2c43aaf2-a973-4b5e-8eba-ad97c89d7555" containerName="main"
May 26 11:42:42.238472 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:42:42.238457    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-b9b644d5d-54xv9"
May 26 11:42:42.243936 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:42:42.243910    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvdde380eaa9fe1facad32d45131f9e34d-kserve-self-signed-certs\""
May 26 11:42:42.261137 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:42:42.261101    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-b9b644d5d-54xv9"]
May 26 11:42:42.329419 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:42:42.329389    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ed85c58-ef71-4fde-b353-88d665ab6b39-home\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-b9b644d5d-54xv9\" (UID: \"1ed85c58-ef71-4fde-b353-88d665ab6b39\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-b9b644d5d-54xv9"
May 26 11:42:42.329588 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:42:42.329458    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ed85c58-ef71-4fde-b353-88d665ab6b39-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-b9b644d5d-54xv9\" (UID: \"1ed85c58-ef71-4fde-b353-88d665ab6b39\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-b9b644d5d-54xv9"
May 26 11:42:42.329588 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:42:42.329494    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ed85c58-ef71-4fde-b353-88d665ab6b39-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-b9b644d5d-54xv9\" (UID: \"1ed85c58-ef71-4fde-b353-88d665ab6b39\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-b9b644d5d-54xv9"
May 26 11:42:42.329588 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:42:42.329518    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvvsg\" (UniqueName: \"kubernetes.io/projected/1ed85c58-ef71-4fde-b353-88d665ab6b39-kube-api-access-wvvsg\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-b9b644d5d-54xv9\" (UID: \"1ed85c58-ef71-4fde-b353-88d665ab6b39\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-b9b644d5d-54xv9"
May 26 11:42:42.329588 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:42:42.329544    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ed85c58-ef71-4fde-b353-88d665ab6b39-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-b9b644d5d-54xv9\" (UID: \"1ed85c58-ef71-4fde-b353-88d665ab6b39\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-b9b644d5d-54xv9"
May 26 11:42:42.329588 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:42:42.329563    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1ed85c58-ef71-4fde-b353-88d665ab6b39-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-b9b644d5d-54xv9\" (UID: \"1ed85c58-ef71-4fde-b353-88d665ab6b39\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-b9b644d5d-54xv9"
May 26 11:42:42.329750 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:42:42.329586    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ed85c58-ef71-4fde-b353-88d665ab6b39-dshm\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-b9b644d5d-54xv9\" (UID: \"1ed85c58-ef71-4fde-b353-88d665ab6b39\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-b9b644d5d-54xv9"
May 26 11:42:42.430580 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:42:42.430547    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ed85c58-ef71-4fde-b353-88d665ab6b39-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-b9b644d5d-54xv9\" (UID: \"1ed85c58-ef71-4fde-b353-88d665ab6b39\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-b9b644d5d-54xv9"
May 26 11:42:42.430580 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:42:42.430584    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1ed85c58-ef71-4fde-b353-88d665ab6b39-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-b9b644d5d-54xv9\" (UID: \"1ed85c58-ef71-4fde-b353-88d665ab6b39\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-b9b644d5d-54xv9"
May 26 11:42:42.430816 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:42:42.430610    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ed85c58-ef71-4fde-b353-88d665ab6b39-dshm\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-b9b644d5d-54xv9\" (UID: \"1ed85c58-ef71-4fde-b353-88d665ab6b39\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-b9b644d5d-54xv9"
May 26 11:42:42.430816 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:42:42.430654    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ed85c58-ef71-4fde-b353-88d665ab6b39-home\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-b9b644d5d-54xv9\" (UID: \"1ed85c58-ef71-4fde-b353-88d665ab6b39\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-b9b644d5d-54xv9"
May 26 11:42:42.430816 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:42:42.430697    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ed85c58-ef71-4fde-b353-88d665ab6b39-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-b9b644d5d-54xv9\" (UID: \"1ed85c58-ef71-4fde-b353-88d665ab6b39\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-b9b644d5d-54xv9"
May 26 11:42:42.430978 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:42:42.430826    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ed85c58-ef71-4fde-b353-88d665ab6b39-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-b9b644d5d-54xv9\" (UID: \"1ed85c58-ef71-4fde-b353-88d665ab6b39\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-b9b644d5d-54xv9"
May 26 11:42:42.430978 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:42:42.430876    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-wvvsg\" (UniqueName: \"kubernetes.io/projected/1ed85c58-ef71-4fde-b353-88d665ab6b39-kube-api-access-wvvsg\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-b9b644d5d-54xv9\" (UID: \"1ed85c58-ef71-4fde-b353-88d665ab6b39\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-b9b644d5d-54xv9"
May 26 11:42:42.430978 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:42:42.430935    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ed85c58-ef71-4fde-b353-88d665ab6b39-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-b9b644d5d-54xv9\" (UID: \"1ed85c58-ef71-4fde-b353-88d665ab6b39\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-b9b644d5d-54xv9"
May 26 11:42:42.431103 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:42:42.431009    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1ed85c58-ef71-4fde-b353-88d665ab6b39-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-b9b644d5d-54xv9\" (UID: \"1ed85c58-ef71-4fde-b353-88d665ab6b39\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-b9b644d5d-54xv9"
May 26 11:42:42.431103 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:42:42.431057    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ed85c58-ef71-4fde-b353-88d665ab6b39-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-b9b644d5d-54xv9\" (UID: \"1ed85c58-ef71-4fde-b353-88d665ab6b39\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-b9b644d5d-54xv9"
May 26 11:42:42.431168 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:42:42.431097    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ed85c58-ef71-4fde-b353-88d665ab6b39-home\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-b9b644d5d-54xv9\" (UID: \"1ed85c58-ef71-4fde-b353-88d665ab6b39\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-b9b644d5d-54xv9"
May 26 11:42:42.433029 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:42:42.433012    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ed85c58-ef71-4fde-b353-88d665ab6b39-dshm\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-b9b644d5d-54xv9\" (UID: \"1ed85c58-ef71-4fde-b353-88d665ab6b39\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-b9b644d5d-54xv9"
May 26 11:42:42.433301 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:42:42.433280    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ed85c58-ef71-4fde-b353-88d665ab6b39-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-b9b644d5d-54xv9\" (UID: \"1ed85c58-ef71-4fde-b353-88d665ab6b39\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-b9b644d5d-54xv9"
May 26 11:42:42.438688 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:42:42.438663    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvvsg\" (UniqueName: \"kubernetes.io/projected/1ed85c58-ef71-4fde-b353-88d665ab6b39-kube-api-access-wvvsg\") pod \"llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-b9b644d5d-54xv9\" (UID: \"1ed85c58-ef71-4fde-b353-88d665ab6b39\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-b9b644d5d-54xv9"
May 26 11:42:42.549331 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:42:42.549229    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-b9b644d5d-54xv9"
May 26 11:42:42.693667 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:42:42.693512    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-b9b644d5d-54xv9"]
May 26 11:42:43.290759 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:42:43.290726    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-b9b644d5d-54xv9" event={"ID":"1ed85c58-ef71-4fde-b353-88d665ab6b39","Type":"ContainerStarted","Data":"5882e40c86d22cff8bf3bb4ee77ba6726aa14ffa7cae7eccbddc378701aa79ad"}
May 26 11:42:43.290759 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:42:43.290763    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-b9b644d5d-54xv9" event={"ID":"1ed85c58-ef71-4fde-b353-88d665ab6b39","Type":"ContainerStarted","Data":"7c4b4b966bff15425dd6319754cad895c11dd2f238daae73980c9c92b49bd691"}
May 26 11:42:54.334120 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:42:54.334040    2572 generic.go:358] "Generic (PLEG): container finished" podID="765046db-e719-4f14-9bbb-15ab16f5c946" containerID="d3acc31fbb8ee1e975209f5d2d299afc29eef49861d995ec4996afddd4d47ea0" exitCode=0
May 26 11:42:54.334624 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:42:54.334114    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-754cf65554-plnxv" event={"ID":"765046db-e719-4f14-9bbb-15ab16f5c946","Type":"ContainerDied","Data":"d3acc31fbb8ee1e975209f5d2d299afc29eef49861d995ec4996afddd4d47ea0"}
May 26 11:42:55.341800 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:42:55.341765    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-754cf65554-plnxv" event={"ID":"765046db-e719-4f14-9bbb-15ab16f5c946","Type":"ContainerStarted","Data":"50862ac75fc1b49e8b579145956938862a14526c48de41edbb25ba02bb049974"}
May 26 11:42:55.365028 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:42:55.364980    2572 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-754cf65554-plnxv" podStartSLOduration=90.473656562 podStartE2EDuration="1m34.364967377s" podCreationTimestamp="2026-05-26 11:41:21 +0000 UTC" firstStartedPulling="2026-05-26 11:41:21.872859203 +0000 UTC m=+1004.775069510" lastFinishedPulling="2026-05-26 11:41:25.764170007 +0000 UTC m=+1008.666380325" observedRunningTime="2026-05-26 11:42:55.362426452 +0000 UTC m=+1098.264636782" watchObservedRunningTime="2026-05-26 11:42:55.364967377 +0000 UTC m=+1098.267177707"
May 26 11:43:01.737476 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:43:01.737447    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-754cf65554-plnxv"
May 26 11:43:01.737476 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:43:01.737484    2572 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-754cf65554-plnxv"
May 26 11:43:01.749443 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:43:01.749423    2572 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-754cf65554-plnxv"
May 26 11:43:02.375686 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:43:02.375654    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-754cf65554-plnxv"
May 26 11:43:03.761108 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:43:03.761078    2572 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-754cf65554-plnxv"]
May 26 11:43:04.371311 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:43:04.371272    2572 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-754cf65554-plnxv" podUID="765046db-e719-4f14-9bbb-15ab16f5c946" containerName="main" containerID="cri-o://50862ac75fc1b49e8b579145956938862a14526c48de41edbb25ba02bb049974" gracePeriod=30
May 26 11:43:04.619345 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:43:04.619323    2572 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-754cf65554-plnxv"
May 26 11:43:04.736160 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:43:04.736130    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h6rfb\" (UniqueName: \"kubernetes.io/projected/765046db-e719-4f14-9bbb-15ab16f5c946-kube-api-access-h6rfb\") pod \"765046db-e719-4f14-9bbb-15ab16f5c946\" (UID: \"765046db-e719-4f14-9bbb-15ab16f5c946\") "
May 26 11:43:04.736372 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:43:04.736199    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/765046db-e719-4f14-9bbb-15ab16f5c946-tmp-dir\") pod \"765046db-e719-4f14-9bbb-15ab16f5c946\" (UID: \"765046db-e719-4f14-9bbb-15ab16f5c946\") "
May 26 11:43:04.736372 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:43:04.736234    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/765046db-e719-4f14-9bbb-15ab16f5c946-model-cache\") pod \"765046db-e719-4f14-9bbb-15ab16f5c946\" (UID: \"765046db-e719-4f14-9bbb-15ab16f5c946\") "
May 26 11:43:04.736372 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:43:04.736265    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/765046db-e719-4f14-9bbb-15ab16f5c946-dshm\") pod \"765046db-e719-4f14-9bbb-15ab16f5c946\" (UID: \"765046db-e719-4f14-9bbb-15ab16f5c946\") "
May 26 11:43:04.736372 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:43:04.736284    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/765046db-e719-4f14-9bbb-15ab16f5c946-home\") pod \"765046db-e719-4f14-9bbb-15ab16f5c946\" (UID: \"765046db-e719-4f14-9bbb-15ab16f5c946\") "
May 26 11:43:04.736372 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:43:04.736312    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/765046db-e719-4f14-9bbb-15ab16f5c946-kserve-provision-location\") pod \"765046db-e719-4f14-9bbb-15ab16f5c946\" (UID: \"765046db-e719-4f14-9bbb-15ab16f5c946\") "
May 26 11:43:04.736372 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:43:04.736340    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/765046db-e719-4f14-9bbb-15ab16f5c946-tls-certs\") pod \"765046db-e719-4f14-9bbb-15ab16f5c946\" (UID: \"765046db-e719-4f14-9bbb-15ab16f5c946\") "
May 26 11:43:04.736627 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:43:04.736430    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/765046db-e719-4f14-9bbb-15ab16f5c946-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "765046db-e719-4f14-9bbb-15ab16f5c946" (UID: "765046db-e719-4f14-9bbb-15ab16f5c946"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 11:43:04.736627 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:43:04.736512    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/765046db-e719-4f14-9bbb-15ab16f5c946-model-cache" (OuterVolumeSpecName: "model-cache") pod "765046db-e719-4f14-9bbb-15ab16f5c946" (UID: "765046db-e719-4f14-9bbb-15ab16f5c946"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 11:43:04.736627 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:43:04.736597    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/765046db-e719-4f14-9bbb-15ab16f5c946-home" (OuterVolumeSpecName: "home") pod "765046db-e719-4f14-9bbb-15ab16f5c946" (UID: "765046db-e719-4f14-9bbb-15ab16f5c946"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 11:43:04.736764 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:43:04.736732    2572 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/765046db-e719-4f14-9bbb-15ab16f5c946-tmp-dir\") on node \"ip-10-0-136-194.ec2.internal\" DevicePath \"\""
May 26 11:43:04.736764 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:43:04.736756    2572 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/765046db-e719-4f14-9bbb-15ab16f5c946-model-cache\") on node \"ip-10-0-136-194.ec2.internal\" DevicePath \"\""
May 26 11:43:04.736834 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:43:04.736774    2572 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/765046db-e719-4f14-9bbb-15ab16f5c946-home\") on node \"ip-10-0-136-194.ec2.internal\" DevicePath \"\""
May 26 11:43:04.738513 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:43:04.738483    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/765046db-e719-4f14-9bbb-15ab16f5c946-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "765046db-e719-4f14-9bbb-15ab16f5c946" (UID: "765046db-e719-4f14-9bbb-15ab16f5c946"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 11:43:04.738682 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:43:04.738662    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/765046db-e719-4f14-9bbb-15ab16f5c946-kube-api-access-h6rfb" (OuterVolumeSpecName: "kube-api-access-h6rfb") pod "765046db-e719-4f14-9bbb-15ab16f5c946" (UID: "765046db-e719-4f14-9bbb-15ab16f5c946"). InnerVolumeSpecName "kube-api-access-h6rfb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 11:43:04.738779 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:43:04.738752    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/765046db-e719-4f14-9bbb-15ab16f5c946-dshm" (OuterVolumeSpecName: "dshm") pod "765046db-e719-4f14-9bbb-15ab16f5c946" (UID: "765046db-e719-4f14-9bbb-15ab16f5c946"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 11:43:04.790911 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:43:04.790880    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/765046db-e719-4f14-9bbb-15ab16f5c946-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "765046db-e719-4f14-9bbb-15ab16f5c946" (UID: "765046db-e719-4f14-9bbb-15ab16f5c946"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 11:43:04.838259 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:43:04.838236    2572 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/765046db-e719-4f14-9bbb-15ab16f5c946-dshm\") on node \"ip-10-0-136-194.ec2.internal\" DevicePath \"\""
May 26 11:43:04.838351 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:43:04.838260    2572 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/765046db-e719-4f14-9bbb-15ab16f5c946-kserve-provision-location\") on node \"ip-10-0-136-194.ec2.internal\" DevicePath \"\""
May 26 11:43:04.838351 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:43:04.838273    2572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/765046db-e719-4f14-9bbb-15ab16f5c946-tls-certs\") on node \"ip-10-0-136-194.ec2.internal\" DevicePath \"\""
May 26 11:43:04.838351 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:43:04.838283    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h6rfb\" (UniqueName: \"kubernetes.io/projected/765046db-e719-4f14-9bbb-15ab16f5c946-kube-api-access-h6rfb\") on node \"ip-10-0-136-194.ec2.internal\" DevicePath \"\""
May 26 11:43:05.376348 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:43:05.376312    2572 generic.go:358] "Generic (PLEG): container finished" podID="765046db-e719-4f14-9bbb-15ab16f5c946" containerID="50862ac75fc1b49e8b579145956938862a14526c48de41edbb25ba02bb049974" exitCode=0
May 26 11:43:05.376516 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:43:05.376381    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-754cf65554-plnxv" event={"ID":"765046db-e719-4f14-9bbb-15ab16f5c946","Type":"ContainerDied","Data":"50862ac75fc1b49e8b579145956938862a14526c48de41edbb25ba02bb049974"}
May 26 11:43:05.376516 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:43:05.376385    2572 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-754cf65554-plnxv"
May 26 11:43:05.376516 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:43:05.376406    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-754cf65554-plnxv" event={"ID":"765046db-e719-4f14-9bbb-15ab16f5c946","Type":"ContainerDied","Data":"a840b7f9e25cec61756b3d799249ccd45565015cf180a6745179c3067ff5300c"}
May 26 11:43:05.376516 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:43:05.376422    2572 scope.go:117] "RemoveContainer" containerID="50862ac75fc1b49e8b579145956938862a14526c48de41edbb25ba02bb049974"
May 26 11:43:05.385220 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:43:05.385178    2572 scope.go:117] "RemoveContainer" containerID="d3acc31fbb8ee1e975209f5d2d299afc29eef49861d995ec4996afddd4d47ea0"
May 26 11:43:05.398748 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:43:05.398726    2572 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-754cf65554-plnxv"]
May 26 11:43:05.399630 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:43:05.399614    2572 scope.go:117] "RemoveContainer" containerID="50862ac75fc1b49e8b579145956938862a14526c48de41edbb25ba02bb049974"
May 26 11:43:05.399867 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:43:05.399848    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50862ac75fc1b49e8b579145956938862a14526c48de41edbb25ba02bb049974\": container with ID starting with 50862ac75fc1b49e8b579145956938862a14526c48de41edbb25ba02bb049974 not found: ID does not exist" containerID="50862ac75fc1b49e8b579145956938862a14526c48de41edbb25ba02bb049974"
May 26 11:43:05.399912 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:43:05.399877    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50862ac75fc1b49e8b579145956938862a14526c48de41edbb25ba02bb049974"} err="failed to get container status \"50862ac75fc1b49e8b579145956938862a14526c48de41edbb25ba02bb049974\": rpc error: code = NotFound desc = could not find container \"50862ac75fc1b49e8b579145956938862a14526c48de41edbb25ba02bb049974\": container with ID starting with 50862ac75fc1b49e8b579145956938862a14526c48de41edbb25ba02bb049974 not found: ID does not exist"
May 26 11:43:05.399912 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:43:05.399895    2572 scope.go:117] "RemoveContainer" containerID="d3acc31fbb8ee1e975209f5d2d299afc29eef49861d995ec4996afddd4d47ea0"
May 26 11:43:05.400100 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:43:05.400081    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3acc31fbb8ee1e975209f5d2d299afc29eef49861d995ec4996afddd4d47ea0\": container with ID starting with d3acc31fbb8ee1e975209f5d2d299afc29eef49861d995ec4996afddd4d47ea0 not found: ID does not exist" containerID="d3acc31fbb8ee1e975209f5d2d299afc29eef49861d995ec4996afddd4d47ea0"
May 26 11:43:05.400138 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:43:05.400107    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3acc31fbb8ee1e975209f5d2d299afc29eef49861d995ec4996afddd4d47ea0"} err="failed to get container status \"d3acc31fbb8ee1e975209f5d2d299afc29eef49861d995ec4996afddd4d47ea0\": rpc error: code = NotFound desc = could not find container \"d3acc31fbb8ee1e975209f5d2d299afc29eef49861d995ec4996afddd4d47ea0\": container with ID starting with d3acc31fbb8ee1e975209f5d2d299afc29eef49861d995ec4996afddd4d47ea0 not found: ID does not exist"
May 26 11:43:05.404199 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:43:05.404165    2572 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-754cf65554-plnxv"]
May 26 11:43:05.675989 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:43:05.675958    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="765046db-e719-4f14-9bbb-15ab16f5c946" path="/var/lib/kubelet/pods/765046db-e719-4f14-9bbb-15ab16f5c946/volumes"
May 26 11:43:09.400748 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:43:09.400719    2572 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-7d4dc488b6-8s2b4"]
May 26 11:43:09.401206 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:43:09.401155    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="765046db-e719-4f14-9bbb-15ab16f5c946" containerName="storage-initializer"
May 26 11:43:09.401206 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:43:09.401169    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="765046db-e719-4f14-9bbb-15ab16f5c946" containerName="storage-initializer"
May 26 11:43:09.401206 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:43:09.401177    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="765046db-e719-4f14-9bbb-15ab16f5c946" containerName="main"
May 26 11:43:09.401206 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:43:09.401182    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="765046db-e719-4f14-9bbb-15ab16f5c946" containerName="main"
May 26 11:43:09.401346 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:43:09.401272    2572 memory_manager.go:356] "RemoveStaleState removing state" podUID="765046db-e719-4f14-9bbb-15ab16f5c946" containerName="main"
May 26 11:43:09.404660 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:43:09.404646    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-7d4dc488b6-8s2b4"
May 26 11:43:09.410487 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:43:09.410448    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-configmap-ref-test-kserve-self-signed-certs\""
May 26 11:43:09.425897 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:43:09.425862    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-7d4dc488b6-8s2b4"]
May 26 11:43:09.479779 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:43:09.479756    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/838e80a5-4f08-4ffe-9010-03252436c89d-model-cache\") pod \"scheduler-configmap-ref-test-kserve-7d4dc488b6-8s2b4\" (UID: \"838e80a5-4f08-4ffe-9010-03252436c89d\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-7d4dc488b6-8s2b4"
May 26 11:43:09.479919 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:43:09.479789    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/838e80a5-4f08-4ffe-9010-03252436c89d-tmp-dir\") pod \"scheduler-configmap-ref-test-kserve-7d4dc488b6-8s2b4\" (UID: \"838e80a5-4f08-4ffe-9010-03252436c89d\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-7d4dc488b6-8s2b4"
May 26 11:43:09.479919 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:43:09.479817    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/838e80a5-4f08-4ffe-9010-03252436c89d-dshm\") pod \"scheduler-configmap-ref-test-kserve-7d4dc488b6-8s2b4\" (UID: \"838e80a5-4f08-4ffe-9010-03252436c89d\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-7d4dc488b6-8s2b4"
May 26 11:43:09.479919 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:43:09.479884    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txslc\" (UniqueName: \"kubernetes.io/projected/838e80a5-4f08-4ffe-9010-03252436c89d-kube-api-access-txslc\") pod \"scheduler-configmap-ref-test-kserve-7d4dc488b6-8s2b4\" (UID: \"838e80a5-4f08-4ffe-9010-03252436c89d\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-7d4dc488b6-8s2b4"
May 26 11:43:09.480092 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:43:09.479966    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/838e80a5-4f08-4ffe-9010-03252436c89d-kserve-provision-location\") pod \"scheduler-configmap-ref-test-kserve-7d4dc488b6-8s2b4\" (UID: \"838e80a5-4f08-4ffe-9010-03252436c89d\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-7d4dc488b6-8s2b4"
May 26 11:43:09.480092 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:43:09.480009    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/838e80a5-4f08-4ffe-9010-03252436c89d-home\") pod \"scheduler-configmap-ref-test-kserve-7d4dc488b6-8s2b4\" (UID: \"838e80a5-4f08-4ffe-9010-03252436c89d\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-7d4dc488b6-8s2b4"
May 26 11:43:09.480092 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:43:09.480037    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/838e80a5-4f08-4ffe-9010-03252436c89d-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-7d4dc488b6-8s2b4\" (UID: \"838e80a5-4f08-4ffe-9010-03252436c89d\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-7d4dc488b6-8s2b4"
May 26 11:43:09.580509 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:43:09.580482    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/838e80a5-4f08-4ffe-9010-03252436c89d-kserve-provision-location\") pod \"scheduler-configmap-ref-test-kserve-7d4dc488b6-8s2b4\" (UID: \"838e80a5-4f08-4ffe-9010-03252436c89d\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-7d4dc488b6-8s2b4"
May 26 11:43:09.580625 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:43:09.580519    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/838e80a5-4f08-4ffe-9010-03252436c89d-home\") pod \"scheduler-configmap-ref-test-kserve-7d4dc488b6-8s2b4\" (UID: \"838e80a5-4f08-4ffe-9010-03252436c89d\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-7d4dc488b6-8s2b4"
May 26 11:43:09.580625 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:43:09.580542    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/838e80a5-4f08-4ffe-9010-03252436c89d-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-7d4dc488b6-8s2b4\" (UID: \"838e80a5-4f08-4ffe-9010-03252436c89d\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-7d4dc488b6-8s2b4"
May 26 11:43:09.580705 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:43:09.580649    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/838e80a5-4f08-4ffe-9010-03252436c89d-model-cache\") pod \"scheduler-configmap-ref-test-kserve-7d4dc488b6-8s2b4\" (UID: \"838e80a5-4f08-4ffe-9010-03252436c89d\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-7d4dc488b6-8s2b4"
May 26 11:43:09.580705 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:43:09.580677    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/838e80a5-4f08-4ffe-9010-03252436c89d-tmp-dir\") pod \"scheduler-configmap-ref-test-kserve-7d4dc488b6-8s2b4\" (UID: \"838e80a5-4f08-4ffe-9010-03252436c89d\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-7d4dc488b6-8s2b4"
May 26 11:43:09.580795 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:43:09.580705    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/838e80a5-4f08-4ffe-9010-03252436c89d-dshm\") pod \"scheduler-configmap-ref-test-kserve-7d4dc488b6-8s2b4\" (UID: \"838e80a5-4f08-4ffe-9010-03252436c89d\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-7d4dc488b6-8s2b4"
May 26 11:43:09.580795 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:43:09.580729    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-txslc\" (UniqueName: \"kubernetes.io/projected/838e80a5-4f08-4ffe-9010-03252436c89d-kube-api-access-txslc\") pod \"scheduler-configmap-ref-test-kserve-7d4dc488b6-8s2b4\" (UID: \"838e80a5-4f08-4ffe-9010-03252436c89d\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-7d4dc488b6-8s2b4"
May 26 11:43:09.580944 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:43:09.580919    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/838e80a5-4f08-4ffe-9010-03252436c89d-home\") pod \"scheduler-configmap-ref-test-kserve-7d4dc488b6-8s2b4\" (UID: \"838e80a5-4f08-4ffe-9010-03252436c89d\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-7d4dc488b6-8s2b4"
May 26 11:43:09.581070 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:43:09.581048    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/838e80a5-4f08-4ffe-9010-03252436c89d-model-cache\") pod \"scheduler-configmap-ref-test-kserve-7d4dc488b6-8s2b4\" (UID: \"838e80a5-4f08-4ffe-9010-03252436c89d\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-7d4dc488b6-8s2b4"
May 26 11:43:09.581129 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:43:09.581072    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/838e80a5-4f08-4ffe-9010-03252436c89d-tmp-dir\") pod \"scheduler-configmap-ref-test-kserve-7d4dc488b6-8s2b4\" (UID: \"838e80a5-4f08-4ffe-9010-03252436c89d\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-7d4dc488b6-8s2b4"
May 26 11:43:09.581223 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:43:09.581174    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/838e80a5-4f08-4ffe-9010-03252436c89d-kserve-provision-location\") pod \"scheduler-configmap-ref-test-kserve-7d4dc488b6-8s2b4\" (UID: \"838e80a5-4f08-4ffe-9010-03252436c89d\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-7d4dc488b6-8s2b4"
May 26 11:43:09.582838 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:43:09.582811    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/838e80a5-4f08-4ffe-9010-03252436c89d-dshm\") pod \"scheduler-configmap-ref-test-kserve-7d4dc488b6-8s2b4\" (UID: \"838e80a5-4f08-4ffe-9010-03252436c89d\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-7d4dc488b6-8s2b4"
May 26 11:43:09.582995 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:43:09.582978    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/838e80a5-4f08-4ffe-9010-03252436c89d-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-7d4dc488b6-8s2b4\" (UID: \"838e80a5-4f08-4ffe-9010-03252436c89d\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-7d4dc488b6-8s2b4"
May 26 11:43:09.589822 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:43:09.589803    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-txslc\" (UniqueName: \"kubernetes.io/projected/838e80a5-4f08-4ffe-9010-03252436c89d-kube-api-access-txslc\") pod \"scheduler-configmap-ref-test-kserve-7d4dc488b6-8s2b4\" (UID: \"838e80a5-4f08-4ffe-9010-03252436c89d\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-7d4dc488b6-8s2b4"
May 26 11:43:09.715885 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:43:09.715855    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-7d4dc488b6-8s2b4"
May 26 11:43:09.850868 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:43:09.850839    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-7d4dc488b6-8s2b4"]
May 26 11:43:09.852992 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:43:09.852965    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod838e80a5_4f08_4ffe_9010_03252436c89d.slice/crio-976c7b1a25b38d4e3e28a2493a6c2867d7aa1ad384e078c65c9580572a958540 WatchSource:0}: Error finding container 976c7b1a25b38d4e3e28a2493a6c2867d7aa1ad384e078c65c9580572a958540: Status 404 returned error can't find the container with id 976c7b1a25b38d4e3e28a2493a6c2867d7aa1ad384e078c65c9580572a958540
May 26 11:43:10.395645 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:43:10.395610    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-7d4dc488b6-8s2b4" event={"ID":"838e80a5-4f08-4ffe-9010-03252436c89d","Type":"ContainerStarted","Data":"f1319a0c7c9047df8fb894245c914f310cae36ad741a8c58c0e968dbfe75a474"}
May 26 11:43:10.395786 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:43:10.395653    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-7d4dc488b6-8s2b4" event={"ID":"838e80a5-4f08-4ffe-9010-03252436c89d","Type":"ContainerStarted","Data":"976c7b1a25b38d4e3e28a2493a6c2867d7aa1ad384e078c65c9580572a958540"}
May 26 11:43:14.411779 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:43:14.411745    2572 generic.go:358] "Generic (PLEG): container finished" podID="838e80a5-4f08-4ffe-9010-03252436c89d" containerID="f1319a0c7c9047df8fb894245c914f310cae36ad741a8c58c0e968dbfe75a474" exitCode=0
May 26 11:43:14.412215 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:43:14.411818    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-7d4dc488b6-8s2b4" event={"ID":"838e80a5-4f08-4ffe-9010-03252436c89d","Type":"ContainerDied","Data":"f1319a0c7c9047df8fb894245c914f310cae36ad741a8c58c0e968dbfe75a474"}
May 26 11:43:15.416971 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:43:15.416930    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-7d4dc488b6-8s2b4" event={"ID":"838e80a5-4f08-4ffe-9010-03252436c89d","Type":"ContainerStarted","Data":"a1b81febc67f92411e75d49126693a17045e92169fb60aadfe894a8f6ee1098b"}
May 26 11:43:15.442928 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:43:15.442881    2572 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-7d4dc488b6-8s2b4" podStartSLOduration=6.442867788 podStartE2EDuration="6.442867788s" podCreationTimestamp="2026-05-26 11:43:09 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 11:43:15.439658963 +0000 UTC m=+1118.341869302" watchObservedRunningTime="2026-05-26 11:43:15.442867788 +0000 UTC m=+1118.345078117"
May 26 11:43:19.716114 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:43:19.716086    2572 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-7d4dc488b6-8s2b4"
May 26 11:43:19.716503 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:43:19.716127    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-7d4dc488b6-8s2b4"
May 26 11:43:19.728921 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:43:19.728900    2572 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-7d4dc488b6-8s2b4"
May 26 11:43:20.446209 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:43:20.446158    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-7d4dc488b6-8s2b4"
May 26 11:43:43.432099 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:43:43.432061    2572 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-7d4dc488b6-8s2b4"]
May 26 11:43:43.432681 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:43:43.432436    2572 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-7d4dc488b6-8s2b4" podUID="838e80a5-4f08-4ffe-9010-03252436c89d" containerName="main" containerID="cri-o://a1b81febc67f92411e75d49126693a17045e92169fb60aadfe894a8f6ee1098b" gracePeriod=30
May 26 11:43:43.668547 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:43:43.668522    2572 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-7d4dc488b6-8s2b4"
May 26 11:43:43.781637 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:43:43.781564    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/838e80a5-4f08-4ffe-9010-03252436c89d-model-cache\") pod \"838e80a5-4f08-4ffe-9010-03252436c89d\" (UID: \"838e80a5-4f08-4ffe-9010-03252436c89d\") "
May 26 11:43:43.781637 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:43:43.781600    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/838e80a5-4f08-4ffe-9010-03252436c89d-home\") pod \"838e80a5-4f08-4ffe-9010-03252436c89d\" (UID: \"838e80a5-4f08-4ffe-9010-03252436c89d\") "
May 26 11:43:43.781637 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:43:43.781621    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/838e80a5-4f08-4ffe-9010-03252436c89d-dshm\") pod \"838e80a5-4f08-4ffe-9010-03252436c89d\" (UID: \"838e80a5-4f08-4ffe-9010-03252436c89d\") "
May 26 11:43:43.781895 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:43:43.781644    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-txslc\" (UniqueName: \"kubernetes.io/projected/838e80a5-4f08-4ffe-9010-03252436c89d-kube-api-access-txslc\") pod \"838e80a5-4f08-4ffe-9010-03252436c89d\" (UID: \"838e80a5-4f08-4ffe-9010-03252436c89d\") "
May 26 11:43:43.781895 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:43:43.781667    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/838e80a5-4f08-4ffe-9010-03252436c89d-tmp-dir\") pod \"838e80a5-4f08-4ffe-9010-03252436c89d\" (UID: \"838e80a5-4f08-4ffe-9010-03252436c89d\") "
May 26 11:43:43.781895 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:43:43.781687    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/838e80a5-4f08-4ffe-9010-03252436c89d-tls-certs\") pod \"838e80a5-4f08-4ffe-9010-03252436c89d\" (UID: \"838e80a5-4f08-4ffe-9010-03252436c89d\") "
May 26 11:43:43.781895 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:43:43.781789    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/838e80a5-4f08-4ffe-9010-03252436c89d-kserve-provision-location\") pod \"838e80a5-4f08-4ffe-9010-03252436c89d\" (UID: \"838e80a5-4f08-4ffe-9010-03252436c89d\") "
May 26 11:43:43.781895 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:43:43.781832    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/838e80a5-4f08-4ffe-9010-03252436c89d-home" (OuterVolumeSpecName: "home") pod "838e80a5-4f08-4ffe-9010-03252436c89d" (UID: "838e80a5-4f08-4ffe-9010-03252436c89d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 11:43:43.781895 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:43:43.781847    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/838e80a5-4f08-4ffe-9010-03252436c89d-model-cache" (OuterVolumeSpecName: "model-cache") pod "838e80a5-4f08-4ffe-9010-03252436c89d" (UID: "838e80a5-4f08-4ffe-9010-03252436c89d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 11:43:43.782182 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:43:43.781993    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/838e80a5-4f08-4ffe-9010-03252436c89d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "838e80a5-4f08-4ffe-9010-03252436c89d" (UID: "838e80a5-4f08-4ffe-9010-03252436c89d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 11:43:43.782182 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:43:43.782124    2572 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/838e80a5-4f08-4ffe-9010-03252436c89d-model-cache\") on node \"ip-10-0-136-194.ec2.internal\" DevicePath \"\""
May 26 11:43:43.782182 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:43:43.782144    2572 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/838e80a5-4f08-4ffe-9010-03252436c89d-home\") on node \"ip-10-0-136-194.ec2.internal\" DevicePath \"\""
May 26 11:43:43.782182 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:43:43.782160    2572 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/838e80a5-4f08-4ffe-9010-03252436c89d-tmp-dir\") on node \"ip-10-0-136-194.ec2.internal\" DevicePath \"\""
May 26 11:43:43.783841 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:43:43.783809    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/838e80a5-4f08-4ffe-9010-03252436c89d-dshm" (OuterVolumeSpecName: "dshm") pod "838e80a5-4f08-4ffe-9010-03252436c89d" (UID: "838e80a5-4f08-4ffe-9010-03252436c89d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 11:43:43.783952 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:43:43.783861    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/838e80a5-4f08-4ffe-9010-03252436c89d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "838e80a5-4f08-4ffe-9010-03252436c89d" (UID: "838e80a5-4f08-4ffe-9010-03252436c89d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 11:43:43.784113 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:43:43.784095    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/838e80a5-4f08-4ffe-9010-03252436c89d-kube-api-access-txslc" (OuterVolumeSpecName: "kube-api-access-txslc") pod "838e80a5-4f08-4ffe-9010-03252436c89d" (UID: "838e80a5-4f08-4ffe-9010-03252436c89d"). InnerVolumeSpecName "kube-api-access-txslc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 11:43:43.838293 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:43:43.838261    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/838e80a5-4f08-4ffe-9010-03252436c89d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "838e80a5-4f08-4ffe-9010-03252436c89d" (UID: "838e80a5-4f08-4ffe-9010-03252436c89d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 11:43:43.883326 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:43:43.883302    2572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/838e80a5-4f08-4ffe-9010-03252436c89d-tls-certs\") on node \"ip-10-0-136-194.ec2.internal\" DevicePath \"\""
May 26 11:43:43.883326 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:43:43.883324    2572 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/838e80a5-4f08-4ffe-9010-03252436c89d-kserve-provision-location\") on node \"ip-10-0-136-194.ec2.internal\" DevicePath \"\""
May 26 11:43:43.883470 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:43:43.883335    2572 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/838e80a5-4f08-4ffe-9010-03252436c89d-dshm\") on node \"ip-10-0-136-194.ec2.internal\" DevicePath \"\""
May 26 11:43:43.883470 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:43:43.883345    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-txslc\" (UniqueName: \"kubernetes.io/projected/838e80a5-4f08-4ffe-9010-03252436c89d-kube-api-access-txslc\") on node \"ip-10-0-136-194.ec2.internal\" DevicePath \"\""
May 26 11:43:44.518603 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:43:44.518568    2572 generic.go:358] "Generic (PLEG): container finished" podID="838e80a5-4f08-4ffe-9010-03252436c89d" containerID="a1b81febc67f92411e75d49126693a17045e92169fb60aadfe894a8f6ee1098b" exitCode=0
May 26 11:43:44.519049 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:43:44.518634    2572 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-7d4dc488b6-8s2b4"
May 26 11:43:44.519049 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:43:44.518656    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-7d4dc488b6-8s2b4" event={"ID":"838e80a5-4f08-4ffe-9010-03252436c89d","Type":"ContainerDied","Data":"a1b81febc67f92411e75d49126693a17045e92169fb60aadfe894a8f6ee1098b"}
May 26 11:43:44.519049 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:43:44.518705    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-7d4dc488b6-8s2b4" event={"ID":"838e80a5-4f08-4ffe-9010-03252436c89d","Type":"ContainerDied","Data":"976c7b1a25b38d4e3e28a2493a6c2867d7aa1ad384e078c65c9580572a958540"}
May 26 11:43:44.519049 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:43:44.518728    2572 scope.go:117] "RemoveContainer" containerID="a1b81febc67f92411e75d49126693a17045e92169fb60aadfe894a8f6ee1098b"
May 26 11:43:44.527935 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:43:44.527917    2572 scope.go:117] "RemoveContainer" containerID="f1319a0c7c9047df8fb894245c914f310cae36ad741a8c58c0e968dbfe75a474"
May 26 11:43:44.538654 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:43:44.538635    2572 scope.go:117] "RemoveContainer" containerID="a1b81febc67f92411e75d49126693a17045e92169fb60aadfe894a8f6ee1098b"
May 26 11:43:44.538903 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:43:44.538879    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1b81febc67f92411e75d49126693a17045e92169fb60aadfe894a8f6ee1098b\": container with ID starting with a1b81febc67f92411e75d49126693a17045e92169fb60aadfe894a8f6ee1098b not found: ID does not exist" containerID="a1b81febc67f92411e75d49126693a17045e92169fb60aadfe894a8f6ee1098b"
May 26 11:43:44.539079 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:43:44.538917    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1b81febc67f92411e75d49126693a17045e92169fb60aadfe894a8f6ee1098b"} err="failed to get container status \"a1b81febc67f92411e75d49126693a17045e92169fb60aadfe894a8f6ee1098b\": rpc error: code = NotFound desc = could not find container \"a1b81febc67f92411e75d49126693a17045e92169fb60aadfe894a8f6ee1098b\": container with ID starting with a1b81febc67f92411e75d49126693a17045e92169fb60aadfe894a8f6ee1098b not found: ID does not exist"
May 26 11:43:44.539079 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:43:44.538937    2572 scope.go:117] "RemoveContainer" containerID="f1319a0c7c9047df8fb894245c914f310cae36ad741a8c58c0e968dbfe75a474"
May 26 11:43:44.539335 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:43:44.539294    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1319a0c7c9047df8fb894245c914f310cae36ad741a8c58c0e968dbfe75a474\": container with ID starting with f1319a0c7c9047df8fb894245c914f310cae36ad741a8c58c0e968dbfe75a474 not found: ID does not exist" containerID="f1319a0c7c9047df8fb894245c914f310cae36ad741a8c58c0e968dbfe75a474"
May 26 11:43:44.539398 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:43:44.539328    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1319a0c7c9047df8fb894245c914f310cae36ad741a8c58c0e968dbfe75a474"} err="failed to get container status \"f1319a0c7c9047df8fb894245c914f310cae36ad741a8c58c0e968dbfe75a474\": rpc error: code = NotFound desc = could not find container \"f1319a0c7c9047df8fb894245c914f310cae36ad741a8c58c0e968dbfe75a474\": container with ID starting with f1319a0c7c9047df8fb894245c914f310cae36ad741a8c58c0e968dbfe75a474 not found: ID does not exist"
May 26 11:43:44.541923 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:43:44.541905    2572 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-7d4dc488b6-8s2b4"]
May 26 11:43:44.544871 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:43:44.544851    2572 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-7d4dc488b6-8s2b4"]
May 26 11:43:45.675326 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:43:45.675294    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="838e80a5-4f08-4ffe-9010-03252436c89d" path="/var/lib/kubelet/pods/838e80a5-4f08-4ffe-9010-03252436c89d/volumes"
May 26 11:43:54.340730 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:43:54.340649    2572 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-855b4f5774-tp6z7"]
May 26 11:43:54.341076 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:43:54.341010    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="838e80a5-4f08-4ffe-9010-03252436c89d" containerName="main"
May 26 11:43:54.341076 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:43:54.341020    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="838e80a5-4f08-4ffe-9010-03252436c89d" containerName="main"
May 26 11:43:54.341076 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:43:54.341036    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="838e80a5-4f08-4ffe-9010-03252436c89d" containerName="storage-initializer"
May 26 11:43:54.341076 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:43:54.341042    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="838e80a5-4f08-4ffe-9010-03252436c89d" containerName="storage-initializer"
May 26 11:43:54.341237 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:43:54.341113    2572 memory_manager.go:356] "RemoveStaleState removing state" podUID="838e80a5-4f08-4ffe-9010-03252436c89d" containerName="main"
May 26 11:43:54.344355 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:43:54.344336    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-855b4f5774-tp6z7"
May 26 11:43:54.346813 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:43:54.346795    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-ha-replicas-test-kserve-self-signed-certs\""
May 26 11:43:54.353178 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:43:54.353149    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-855b4f5774-tp6z7"]
May 26 11:43:54.377431 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:43:54.377407    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/031b53f5-7e1a-4a15-82d2-b6695d1f32eb-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-855b4f5774-tp6z7\" (UID: \"031b53f5-7e1a-4a15-82d2-b6695d1f32eb\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-855b4f5774-tp6z7"
May 26 11:43:54.377578 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:43:54.377451    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mcns6\" (UniqueName: \"kubernetes.io/projected/031b53f5-7e1a-4a15-82d2-b6695d1f32eb-kube-api-access-mcns6\") pod \"scheduler-ha-replicas-test-kserve-855b4f5774-tp6z7\" (UID: \"031b53f5-7e1a-4a15-82d2-b6695d1f32eb\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-855b4f5774-tp6z7"
May 26 11:43:54.377578 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:43:54.377478    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/031b53f5-7e1a-4a15-82d2-b6695d1f32eb-dshm\") pod \"scheduler-ha-replicas-test-kserve-855b4f5774-tp6z7\" (UID: \"031b53f5-7e1a-4a15-82d2-b6695d1f32eb\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-855b4f5774-tp6z7"
May 26 11:43:54.377693 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:43:54.377575    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/031b53f5-7e1a-4a15-82d2-b6695d1f32eb-kserve-provision-location\") pod \"scheduler-ha-replicas-test-kserve-855b4f5774-tp6z7\" (UID: \"031b53f5-7e1a-4a15-82d2-b6695d1f32eb\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-855b4f5774-tp6z7"
May 26 11:43:54.377693 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:43:54.377638    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/031b53f5-7e1a-4a15-82d2-b6695d1f32eb-home\") pod \"scheduler-ha-replicas-test-kserve-855b4f5774-tp6z7\" (UID: \"031b53f5-7e1a-4a15-82d2-b6695d1f32eb\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-855b4f5774-tp6z7"
May 26 11:43:54.377805 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:43:54.377714    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/031b53f5-7e1a-4a15-82d2-b6695d1f32eb-model-cache\") pod \"scheduler-ha-replicas-test-kserve-855b4f5774-tp6z7\" (UID: \"031b53f5-7e1a-4a15-82d2-b6695d1f32eb\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-855b4f5774-tp6z7"
May 26 11:43:54.377805 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:43:54.377744    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/031b53f5-7e1a-4a15-82d2-b6695d1f32eb-tmp-dir\") pod \"scheduler-ha-replicas-test-kserve-855b4f5774-tp6z7\" (UID: \"031b53f5-7e1a-4a15-82d2-b6695d1f32eb\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-855b4f5774-tp6z7"
May 26 11:43:54.479028 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:43:54.478997    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/031b53f5-7e1a-4a15-82d2-b6695d1f32eb-model-cache\") pod \"scheduler-ha-replicas-test-kserve-855b4f5774-tp6z7\" (UID: \"031b53f5-7e1a-4a15-82d2-b6695d1f32eb\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-855b4f5774-tp6z7"
May 26 11:43:54.479174 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:43:54.479034    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/031b53f5-7e1a-4a15-82d2-b6695d1f32eb-tmp-dir\") pod \"scheduler-ha-replicas-test-kserve-855b4f5774-tp6z7\" (UID: \"031b53f5-7e1a-4a15-82d2-b6695d1f32eb\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-855b4f5774-tp6z7"
May 26 11:43:54.479174 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:43:54.479060    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/031b53f5-7e1a-4a15-82d2-b6695d1f32eb-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-855b4f5774-tp6z7\" (UID: \"031b53f5-7e1a-4a15-82d2-b6695d1f32eb\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-855b4f5774-tp6z7"
May 26 11:43:54.479174 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:43:54.479103    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-mcns6\" (UniqueName: \"kubernetes.io/projected/031b53f5-7e1a-4a15-82d2-b6695d1f32eb-kube-api-access-mcns6\") pod \"scheduler-ha-replicas-test-kserve-855b4f5774-tp6z7\" (UID: \"031b53f5-7e1a-4a15-82d2-b6695d1f32eb\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-855b4f5774-tp6z7"
May 26 11:43:54.479174 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:43:54.479141    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/031b53f5-7e1a-4a15-82d2-b6695d1f32eb-dshm\") pod \"scheduler-ha-replicas-test-kserve-855b4f5774-tp6z7\" (UID: \"031b53f5-7e1a-4a15-82d2-b6695d1f32eb\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-855b4f5774-tp6z7"
May 26 11:43:54.479396 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:43:54.479180    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/031b53f5-7e1a-4a15-82d2-b6695d1f32eb-kserve-provision-location\") pod \"scheduler-ha-replicas-test-kserve-855b4f5774-tp6z7\" (UID: \"031b53f5-7e1a-4a15-82d2-b6695d1f32eb\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-855b4f5774-tp6z7"
May 26 11:43:54.479396 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:43:54.479238    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/031b53f5-7e1a-4a15-82d2-b6695d1f32eb-home\") pod \"scheduler-ha-replicas-test-kserve-855b4f5774-tp6z7\" (UID: \"031b53f5-7e1a-4a15-82d2-b6695d1f32eb\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-855b4f5774-tp6z7"
May 26 11:43:54.479496 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:43:54.479454    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/031b53f5-7e1a-4a15-82d2-b6695d1f32eb-model-cache\") pod \"scheduler-ha-replicas-test-kserve-855b4f5774-tp6z7\" (UID: \"031b53f5-7e1a-4a15-82d2-b6695d1f32eb\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-855b4f5774-tp6z7"
May 26 11:43:54.479550 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:43:54.479522    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/031b53f5-7e1a-4a15-82d2-b6695d1f32eb-home\") pod \"scheduler-ha-replicas-test-kserve-855b4f5774-tp6z7\" (UID: \"031b53f5-7e1a-4a15-82d2-b6695d1f32eb\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-855b4f5774-tp6z7"
May 26 11:43:54.479608 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:43:54.479574    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/031b53f5-7e1a-4a15-82d2-b6695d1f32eb-kserve-provision-location\") pod \"scheduler-ha-replicas-test-kserve-855b4f5774-tp6z7\" (UID: \"031b53f5-7e1a-4a15-82d2-b6695d1f32eb\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-855b4f5774-tp6z7"
May 26 11:43:54.479704 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:43:54.479687    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/031b53f5-7e1a-4a15-82d2-b6695d1f32eb-tmp-dir\") pod \"scheduler-ha-replicas-test-kserve-855b4f5774-tp6z7\" (UID: \"031b53f5-7e1a-4a15-82d2-b6695d1f32eb\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-855b4f5774-tp6z7"
May 26 11:43:54.481398 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:43:54.481380    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/031b53f5-7e1a-4a15-82d2-b6695d1f32eb-dshm\") pod \"scheduler-ha-replicas-test-kserve-855b4f5774-tp6z7\" (UID: \"031b53f5-7e1a-4a15-82d2-b6695d1f32eb\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-855b4f5774-tp6z7"
May 26 11:43:54.481591 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:43:54.481575    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/031b53f5-7e1a-4a15-82d2-b6695d1f32eb-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-855b4f5774-tp6z7\" (UID: \"031b53f5-7e1a-4a15-82d2-b6695d1f32eb\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-855b4f5774-tp6z7"
May 26 11:43:54.494000 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:43:54.493968    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mcns6\" (UniqueName: \"kubernetes.io/projected/031b53f5-7e1a-4a15-82d2-b6695d1f32eb-kube-api-access-mcns6\") pod \"scheduler-ha-replicas-test-kserve-855b4f5774-tp6z7\" (UID: \"031b53f5-7e1a-4a15-82d2-b6695d1f32eb\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-855b4f5774-tp6z7"
May 26 11:43:54.654936 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:43:54.654903    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-855b4f5774-tp6z7"
May 26 11:43:54.743477 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:43:54.743447    2572 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-747455452ls6"]
May 26 11:43:54.747446 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:43:54.747427    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-747455452ls6"
May 26 11:43:54.750570 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:43:54.750367    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-ha-replicas-test-epp-sa-dockercfg-qh6t9\""
May 26 11:43:54.758461 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:43:54.758438    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-747455452ls6"]
May 26 11:43:54.782472 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:43:54.782444    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/f71a756c-64ed-4d91-90d9-c27281c67dda-tokenizer-uds\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-747455452ls6\" (UID: \"f71a756c-64ed-4d91-90d9-c27281c67dda\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-747455452ls6"
May 26 11:43:54.782620 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:43:54.782478    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f71a756c-64ed-4d91-90d9-c27281c67dda-kserve-provision-location\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-747455452ls6\" (UID: \"f71a756c-64ed-4d91-90d9-c27281c67dda\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-747455452ls6"
May 26 11:43:54.782620 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:43:54.782511    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f71a756c-64ed-4d91-90d9-c27281c67dda-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-747455452ls6\" (UID: \"f71a756c-64ed-4d91-90d9-c27281c67dda\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-747455452ls6"
May 26 11:43:54.782620 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:43:54.782599    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bm2rx\" (UniqueName: \"kubernetes.io/projected/f71a756c-64ed-4d91-90d9-c27281c67dda-kube-api-access-bm2rx\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-747455452ls6\" (UID: \"f71a756c-64ed-4d91-90d9-c27281c67dda\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-747455452ls6"
May 26 11:43:54.782796 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:43:54.782637    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/f71a756c-64ed-4d91-90d9-c27281c67dda-tokenizer-tmp\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-747455452ls6\" (UID: \"f71a756c-64ed-4d91-90d9-c27281c67dda\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-747455452ls6"
May 26 11:43:54.782796 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:43:54.782673    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/f71a756c-64ed-4d91-90d9-c27281c67dda-tokenizer-cache\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-747455452ls6\" (UID: \"f71a756c-64ed-4d91-90d9-c27281c67dda\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-747455452ls6"
May 26 11:43:54.796422 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:43:54.796400    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-855b4f5774-tp6z7"]
May 26 11:43:54.797943 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:43:54.797918    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod031b53f5_7e1a_4a15_82d2_b6695d1f32eb.slice/crio-a3960eceaa2bee25df5937de96f17801b3840268e7c3e775428c35ae9123eec2 WatchSource:0}: Error finding container a3960eceaa2bee25df5937de96f17801b3840268e7c3e775428c35ae9123eec2: Status 404 returned error can't find the container with id a3960eceaa2bee25df5937de96f17801b3840268e7c3e775428c35ae9123eec2
May 26 11:43:54.883941 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:43:54.883908    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-bm2rx\" (UniqueName: \"kubernetes.io/projected/f71a756c-64ed-4d91-90d9-c27281c67dda-kube-api-access-bm2rx\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-747455452ls6\" (UID: \"f71a756c-64ed-4d91-90d9-c27281c67dda\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-747455452ls6"
May 26 11:43:54.884062 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:43:54.883951    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/f71a756c-64ed-4d91-90d9-c27281c67dda-tokenizer-tmp\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-747455452ls6\" (UID: \"f71a756c-64ed-4d91-90d9-c27281c67dda\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-747455452ls6"
May 26 11:43:54.884062 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:43:54.883979    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/f71a756c-64ed-4d91-90d9-c27281c67dda-tokenizer-cache\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-747455452ls6\" (UID: \"f71a756c-64ed-4d91-90d9-c27281c67dda\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-747455452ls6"
May 26 11:43:54.884062 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:43:54.884041    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/f71a756c-64ed-4d91-90d9-c27281c67dda-tokenizer-uds\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-747455452ls6\" (UID: \"f71a756c-64ed-4d91-90d9-c27281c67dda\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-747455452ls6"
May 26 11:43:54.884259 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:43:54.884062    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f71a756c-64ed-4d91-90d9-c27281c67dda-kserve-provision-location\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-747455452ls6\" (UID: \"f71a756c-64ed-4d91-90d9-c27281c67dda\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-747455452ls6"
May 26 11:43:54.884259 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:43:54.884095    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f71a756c-64ed-4d91-90d9-c27281c67dda-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-747455452ls6\" (UID: \"f71a756c-64ed-4d91-90d9-c27281c67dda\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-747455452ls6"
May 26 11:43:54.884435 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:43:54.884411    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/f71a756c-64ed-4d91-90d9-c27281c67dda-tokenizer-tmp\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-747455452ls6\" (UID: \"f71a756c-64ed-4d91-90d9-c27281c67dda\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-747455452ls6"
May 26 11:43:54.884500 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:43:54.884416    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/f71a756c-64ed-4d91-90d9-c27281c67dda-tokenizer-uds\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-747455452ls6\" (UID: \"f71a756c-64ed-4d91-90d9-c27281c67dda\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-747455452ls6"
May 26 11:43:54.884500 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:43:54.884480    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f71a756c-64ed-4d91-90d9-c27281c67dda-kserve-provision-location\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-747455452ls6\" (UID: \"f71a756c-64ed-4d91-90d9-c27281c67dda\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-747455452ls6"
May 26 11:43:54.884591 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:43:54.884511    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/f71a756c-64ed-4d91-90d9-c27281c67dda-tokenizer-cache\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-747455452ls6\" (UID: \"f71a756c-64ed-4d91-90d9-c27281c67dda\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-747455452ls6"
May 26 11:43:54.886360 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:43:54.886345    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f71a756c-64ed-4d91-90d9-c27281c67dda-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-747455452ls6\" (UID: \"f71a756c-64ed-4d91-90d9-c27281c67dda\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-747455452ls6"
May 26 11:43:54.892754 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:43:54.892737    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bm2rx\" (UniqueName: \"kubernetes.io/projected/f71a756c-64ed-4d91-90d9-c27281c67dda-kube-api-access-bm2rx\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-747455452ls6\" (UID: \"f71a756c-64ed-4d91-90d9-c27281c67dda\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-747455452ls6"
May 26 11:43:55.060306 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:43:55.060199    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-747455452ls6"
May 26 11:43:55.200039 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:43:55.200016    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-747455452ls6"]
May 26 11:43:55.202741 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:43:55.202707    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf71a756c_64ed_4d91_90d9_c27281c67dda.slice/crio-79099299b4b5ad7611dc5185ec9640e2912c3d032bb62fdc39caa25165952738 WatchSource:0}: Error finding container 79099299b4b5ad7611dc5185ec9640e2912c3d032bb62fdc39caa25165952738: Status 404 returned error can't find the container with id 79099299b4b5ad7611dc5185ec9640e2912c3d032bb62fdc39caa25165952738
May 26 11:43:55.559214 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:43:55.559166    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-747455452ls6" event={"ID":"f71a756c-64ed-4d91-90d9-c27281c67dda","Type":"ContainerStarted","Data":"4a0d697bfb1b5cd406a987e9defccc1305ba3c7062ba03623e19fe3a610e653a"}
May 26 11:43:55.559214 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:43:55.559219    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-747455452ls6" event={"ID":"f71a756c-64ed-4d91-90d9-c27281c67dda","Type":"ContainerStarted","Data":"79099299b4b5ad7611dc5185ec9640e2912c3d032bb62fdc39caa25165952738"}
May 26 11:43:55.560633 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:43:55.560609    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-855b4f5774-tp6z7" event={"ID":"031b53f5-7e1a-4a15-82d2-b6695d1f32eb","Type":"ContainerStarted","Data":"8834b4a791cdd3db0ef2da1cbf852a858280f117525baa6fb788cabf4e9de6d3"}
May 26 11:43:55.560719 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:43:55.560637    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-855b4f5774-tp6z7" event={"ID":"031b53f5-7e1a-4a15-82d2-b6695d1f32eb","Type":"ContainerStarted","Data":"a3960eceaa2bee25df5937de96f17801b3840268e7c3e775428c35ae9123eec2"}
May 26 11:43:56.565496 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:43:56.565458    2572 generic.go:358] "Generic (PLEG): container finished" podID="f71a756c-64ed-4d91-90d9-c27281c67dda" containerID="4a0d697bfb1b5cd406a987e9defccc1305ba3c7062ba03623e19fe3a610e653a" exitCode=0
May 26 11:43:56.565855 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:43:56.565539    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-747455452ls6" event={"ID":"f71a756c-64ed-4d91-90d9-c27281c67dda","Type":"ContainerDied","Data":"4a0d697bfb1b5cd406a987e9defccc1305ba3c7062ba03623e19fe3a610e653a"}
May 26 11:43:58.577897 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:43:58.577842    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-747455452ls6" event={"ID":"f71a756c-64ed-4d91-90d9-c27281c67dda","Type":"ContainerStarted","Data":"988498d845e1368772fe4eed4d7785ac80ffa9df95a44074af870456e95081b8"}
May 26 11:44:26.697077 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:44:26.697043    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-747455452ls6" event={"ID":"f71a756c-64ed-4d91-90d9-c27281c67dda","Type":"ContainerStarted","Data":"b4f452eadfdd49304c858d322e8ee0e1801878cb9003e4cec9b029d68ca7c2ad"}
May 26 11:44:26.697512 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:44:26.697157    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-747455452ls6"
May 26 11:44:26.719727 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:44:26.719686    2572 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-747455452ls6" podStartSLOduration=3.161395423 podStartE2EDuration="32.719674114s" podCreationTimestamp="2026-05-26 11:43:54 +0000 UTC" firstStartedPulling="2026-05-26 11:43:56.566666347 +0000 UTC m=+1159.468876653" lastFinishedPulling="2026-05-26 11:44:26.124945037 +0000 UTC m=+1189.027155344" observedRunningTime="2026-05-26 11:44:26.718214004 +0000 UTC m=+1189.620424330" watchObservedRunningTime="2026-05-26 11:44:26.719674114 +0000 UTC m=+1189.621884443"
May 26 11:44:27.702660 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:44:27.702621    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-747455452ls6" podUID="f71a756c-64ed-4d91-90d9-c27281c67dda" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
May 26 11:44:35.061261 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:44:35.061225    2572 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-747455452ls6"
May 26 11:44:35.061691 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:44:35.061281    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-747455452ls6"
May 26 11:44:35.062927 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:44:35.062900    2572 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-747455452ls6"
May 26 11:44:35.063073 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:44:35.062914    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-747455452ls6" podUID="f71a756c-64ed-4d91-90d9-c27281c67dda" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
May 26 11:44:35.734264 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:44:35.734232    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-747455452ls6"
May 26 11:44:35.734464 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:44:35.734340    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-747455452ls6" podUID="f71a756c-64ed-4d91-90d9-c27281c67dda" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
May 26 11:44:36.737821 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:44:36.737781    2572 generic.go:358] "Generic (PLEG): container finished" podID="1ed85c58-ef71-4fde-b353-88d665ab6b39" containerID="5882e40c86d22cff8bf3bb4ee77ba6726aa14ffa7cae7eccbddc378701aa79ad" exitCode=0
May 26 11:44:36.738241 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:44:36.737852    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-b9b644d5d-54xv9" event={"ID":"1ed85c58-ef71-4fde-b353-88d665ab6b39","Type":"ContainerDied","Data":"5882e40c86d22cff8bf3bb4ee77ba6726aa14ffa7cae7eccbddc378701aa79ad"}
May 26 11:44:36.739533 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:44:36.739509    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-747455452ls6" podUID="f71a756c-64ed-4d91-90d9-c27281c67dda" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
May 26 11:44:37.647427 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:44:37.647394    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-5bfgg_c6373342-dd07-4497-a4f3-978ea9613b6e/console-operator/2.log"
May 26 11:44:37.649787 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:44:37.649652    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-5bfgg_c6373342-dd07-4497-a4f3-978ea9613b6e/console-operator/2.log"
May 26 11:44:37.657078 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:44:37.657053    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-js8cc_7264c907-f536-4719-905a-2bbda9e3c336/ovn-acl-logging/0.log"
May 26 11:44:37.659575 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:44:37.659554    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-js8cc_7264c907-f536-4719-905a-2bbda9e3c336/ovn-acl-logging/0.log"
May 26 11:44:46.740386 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:44:46.740349    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-747455452ls6" podUID="f71a756c-64ed-4d91-90d9-c27281c67dda" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
May 26 11:44:56.739436 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:44:56.739397    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-747455452ls6" podUID="f71a756c-64ed-4d91-90d9-c27281c67dda" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
May 26 11:45:06.739850 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:45:06.739796    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-747455452ls6" podUID="f71a756c-64ed-4d91-90d9-c27281c67dda" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
May 26 11:45:16.740633 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:45:16.740586    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-747455452ls6" podUID="f71a756c-64ed-4d91-90d9-c27281c67dda" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
May 26 11:45:16.897594 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:45:16.897552    2572 generic.go:358] "Generic (PLEG): container finished" podID="031b53f5-7e1a-4a15-82d2-b6695d1f32eb" containerID="8834b4a791cdd3db0ef2da1cbf852a858280f117525baa6fb788cabf4e9de6d3" exitCode=0
May 26 11:45:16.897776 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:45:16.897628    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-855b4f5774-tp6z7" event={"ID":"031b53f5-7e1a-4a15-82d2-b6695d1f32eb","Type":"ContainerDied","Data":"8834b4a791cdd3db0ef2da1cbf852a858280f117525baa6fb788cabf4e9de6d3"}
May 26 11:45:17.902927 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:45:17.902881    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-b9b644d5d-54xv9" event={"ID":"1ed85c58-ef71-4fde-b353-88d665ab6b39","Type":"ContainerStarted","Data":"18c6e788cac9ba701a92001b7bf4cd9a11b0ff7bd1d9849a4277a49a93316783"}
May 26 11:45:17.904758 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:45:17.904732    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-855b4f5774-tp6z7" event={"ID":"031b53f5-7e1a-4a15-82d2-b6695d1f32eb","Type":"ContainerStarted","Data":"e7689fecc5b094f8ad74e467e750e31a9612c06ab731ba34fc49958d46d34703"}
May 26 11:45:17.926382 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:45:17.926283    2572 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-b9b644d5d-54xv9" podStartSLOduration=115.373057148 podStartE2EDuration="2m35.926262842s" podCreationTimestamp="2026-05-26 11:42:42 +0000 UTC" firstStartedPulling="2026-05-26 11:44:36.739264827 +0000 UTC m=+1199.641475137" lastFinishedPulling="2026-05-26 11:45:17.292470505 +0000 UTC m=+1240.194680831" observedRunningTime="2026-05-26 11:45:17.923134255 +0000 UTC m=+1240.825344624" watchObservedRunningTime="2026-05-26 11:45:17.926262842 +0000 UTC m=+1240.828473172"
May 26 11:45:17.942106 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:45:17.942060    2572 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-855b4f5774-tp6z7" podStartSLOduration=83.942041919 podStartE2EDuration="1m23.942041919s" podCreationTimestamp="2026-05-26 11:43:54 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 11:45:17.940706346 +0000 UTC m=+1240.842916675" watchObservedRunningTime="2026-05-26 11:45:17.942041919 +0000 UTC m=+1240.844252248"
May 26 11:45:22.550156 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:45:22.550109    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-b9b644d5d-54xv9"
May 26 11:45:22.550156 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:45:22.550161    2572 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-b9b644d5d-54xv9"
May 26 11:45:22.551510 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:45:22.551480    2572 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-b9b644d5d-54xv9" podUID="1ed85c58-ef71-4fde-b353-88d665ab6b39" containerName="main" probeResult="failure" output="Get \"https://10.132.0.37:8000/health\": dial tcp 10.132.0.37:8000: connect: connection refused"
May 26 11:45:24.655160 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:45:24.655069    2572 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-855b4f5774-tp6z7"
May 26 11:45:24.655160 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:45:24.655110    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-855b4f5774-tp6z7"
May 26 11:45:24.667978 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:45:24.667955    2572 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-855b4f5774-tp6z7"
May 26 11:45:24.942498 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:45:24.942421    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-855b4f5774-tp6z7"
May 26 11:45:26.452947 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:45:26.452908    2572 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-747455452ls6"]
May 26 11:45:26.453444 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:45:26.453337    2572 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-747455452ls6" podUID="f71a756c-64ed-4d91-90d9-c27281c67dda" containerName="main" containerID="cri-o://988498d845e1368772fe4eed4d7785ac80ffa9df95a44074af870456e95081b8" gracePeriod=30
May 26 11:45:26.453530 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:45:26.453506    2572 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-747455452ls6" podUID="f71a756c-64ed-4d91-90d9-c27281c67dda" containerName="tokenizer" containerID="cri-o://b4f452eadfdd49304c858d322e8ee0e1801878cb9003e4cec9b029d68ca7c2ad" gracePeriod=30
May 26 11:45:26.458418 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:45:26.458393    2572 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-855b4f5774-tp6z7"]
May 26 11:45:26.458947 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:45:26.458918    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-747455452ls6" podUID="f71a756c-64ed-4d91-90d9-c27281c67dda" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
May 26 11:45:26.938782 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:45:26.938748    2572 generic.go:358] "Generic (PLEG): container finished" podID="f71a756c-64ed-4d91-90d9-c27281c67dda" containerID="988498d845e1368772fe4eed4d7785ac80ffa9df95a44074af870456e95081b8" exitCode=0
May 26 11:45:26.938946 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:45:26.938804    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-747455452ls6" event={"ID":"f71a756c-64ed-4d91-90d9-c27281c67dda","Type":"ContainerDied","Data":"988498d845e1368772fe4eed4d7785ac80ffa9df95a44074af870456e95081b8"}
May 26 11:45:26.939076 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:45:26.939053    2572 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-855b4f5774-tp6z7" podUID="031b53f5-7e1a-4a15-82d2-b6695d1f32eb" containerName="main" containerID="cri-o://e7689fecc5b094f8ad74e467e750e31a9612c06ab731ba34fc49958d46d34703" gracePeriod=30
May 26 11:45:27.217333 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:45:27.217302    2572 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-855b4f5774-tp6z7"
May 26 11:45:27.308524 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:45:27.308491    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/031b53f5-7e1a-4a15-82d2-b6695d1f32eb-kserve-provision-location\") pod \"031b53f5-7e1a-4a15-82d2-b6695d1f32eb\" (UID: \"031b53f5-7e1a-4a15-82d2-b6695d1f32eb\") "
May 26 11:45:27.308720 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:45:27.308553    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/031b53f5-7e1a-4a15-82d2-b6695d1f32eb-tls-certs\") pod \"031b53f5-7e1a-4a15-82d2-b6695d1f32eb\" (UID: \"031b53f5-7e1a-4a15-82d2-b6695d1f32eb\") "
May 26 11:45:27.308720 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:45:27.308581    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/031b53f5-7e1a-4a15-82d2-b6695d1f32eb-home\") pod \"031b53f5-7e1a-4a15-82d2-b6695d1f32eb\" (UID: \"031b53f5-7e1a-4a15-82d2-b6695d1f32eb\") "
May 26 11:45:27.308720 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:45:27.308604    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/031b53f5-7e1a-4a15-82d2-b6695d1f32eb-tmp-dir\") pod \"031b53f5-7e1a-4a15-82d2-b6695d1f32eb\" (UID: \"031b53f5-7e1a-4a15-82d2-b6695d1f32eb\") "
May 26 11:45:27.308720 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:45:27.308638    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mcns6\" (UniqueName: \"kubernetes.io/projected/031b53f5-7e1a-4a15-82d2-b6695d1f32eb-kube-api-access-mcns6\") pod \"031b53f5-7e1a-4a15-82d2-b6695d1f32eb\" (UID: \"031b53f5-7e1a-4a15-82d2-b6695d1f32eb\") "
May 26 11:45:27.308937 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:45:27.308850    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/031b53f5-7e1a-4a15-82d2-b6695d1f32eb-home" (OuterVolumeSpecName: "home") pod "031b53f5-7e1a-4a15-82d2-b6695d1f32eb" (UID: "031b53f5-7e1a-4a15-82d2-b6695d1f32eb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 11:45:27.309038 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:45:27.309015    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/031b53f5-7e1a-4a15-82d2-b6695d1f32eb-model-cache\") pod \"031b53f5-7e1a-4a15-82d2-b6695d1f32eb\" (UID: \"031b53f5-7e1a-4a15-82d2-b6695d1f32eb\") "
May 26 11:45:27.309096 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:45:27.309079    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/031b53f5-7e1a-4a15-82d2-b6695d1f32eb-dshm\") pod \"031b53f5-7e1a-4a15-82d2-b6695d1f32eb\" (UID: \"031b53f5-7e1a-4a15-82d2-b6695d1f32eb\") "
May 26 11:45:27.309152 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:45:27.309022    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/031b53f5-7e1a-4a15-82d2-b6695d1f32eb-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "031b53f5-7e1a-4a15-82d2-b6695d1f32eb" (UID: "031b53f5-7e1a-4a15-82d2-b6695d1f32eb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 11:45:27.309235 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:45:27.309151    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/031b53f5-7e1a-4a15-82d2-b6695d1f32eb-model-cache" (OuterVolumeSpecName: "model-cache") pod "031b53f5-7e1a-4a15-82d2-b6695d1f32eb" (UID: "031b53f5-7e1a-4a15-82d2-b6695d1f32eb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 11:45:27.309454 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:45:27.309435    2572 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/031b53f5-7e1a-4a15-82d2-b6695d1f32eb-home\") on node \"ip-10-0-136-194.ec2.internal\" DevicePath \"\""
May 26 11:45:27.309524 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:45:27.309459    2572 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/031b53f5-7e1a-4a15-82d2-b6695d1f32eb-tmp-dir\") on node \"ip-10-0-136-194.ec2.internal\" DevicePath \"\""
May 26 11:45:27.309524 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:45:27.309475    2572 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/031b53f5-7e1a-4a15-82d2-b6695d1f32eb-model-cache\") on node \"ip-10-0-136-194.ec2.internal\" DevicePath \"\""
May 26 11:45:27.311015 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:45:27.310980    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/031b53f5-7e1a-4a15-82d2-b6695d1f32eb-kube-api-access-mcns6" (OuterVolumeSpecName: "kube-api-access-mcns6") pod "031b53f5-7e1a-4a15-82d2-b6695d1f32eb" (UID: "031b53f5-7e1a-4a15-82d2-b6695d1f32eb"). InnerVolumeSpecName "kube-api-access-mcns6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 11:45:27.311623 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:45:27.311597    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/031b53f5-7e1a-4a15-82d2-b6695d1f32eb-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "031b53f5-7e1a-4a15-82d2-b6695d1f32eb" (UID: "031b53f5-7e1a-4a15-82d2-b6695d1f32eb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 11:45:27.311736 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:45:27.311721    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/031b53f5-7e1a-4a15-82d2-b6695d1f32eb-dshm" (OuterVolumeSpecName: "dshm") pod "031b53f5-7e1a-4a15-82d2-b6695d1f32eb" (UID: "031b53f5-7e1a-4a15-82d2-b6695d1f32eb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 11:45:27.363097 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:45:27.363037    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/031b53f5-7e1a-4a15-82d2-b6695d1f32eb-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "031b53f5-7e1a-4a15-82d2-b6695d1f32eb" (UID: "031b53f5-7e1a-4a15-82d2-b6695d1f32eb"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 11:45:27.416220 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:45:27.413158    2572 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/031b53f5-7e1a-4a15-82d2-b6695d1f32eb-kserve-provision-location\") on node \"ip-10-0-136-194.ec2.internal\" DevicePath \"\""
May 26 11:45:27.416220 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:45:27.413215    2572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/031b53f5-7e1a-4a15-82d2-b6695d1f32eb-tls-certs\") on node \"ip-10-0-136-194.ec2.internal\" DevicePath \"\""
May 26 11:45:27.416220 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:45:27.413233    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mcns6\" (UniqueName: \"kubernetes.io/projected/031b53f5-7e1a-4a15-82d2-b6695d1f32eb-kube-api-access-mcns6\") on node \"ip-10-0-136-194.ec2.internal\" DevicePath \"\""
May 26 11:45:27.416220 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:45:27.413254    2572 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/031b53f5-7e1a-4a15-82d2-b6695d1f32eb-dshm\") on node \"ip-10-0-136-194.ec2.internal\" DevicePath \"\""
May 26 11:45:27.739309 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:45:27.739282    2572 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-747455452ls6"
May 26 11:45:27.816677 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:45:27.816596    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/f71a756c-64ed-4d91-90d9-c27281c67dda-tokenizer-cache\") pod \"f71a756c-64ed-4d91-90d9-c27281c67dda\" (UID: \"f71a756c-64ed-4d91-90d9-c27281c67dda\") "
May 26 11:45:27.816677 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:45:27.816657    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f71a756c-64ed-4d91-90d9-c27281c67dda-kserve-provision-location\") pod \"f71a756c-64ed-4d91-90d9-c27281c67dda\" (UID: \"f71a756c-64ed-4d91-90d9-c27281c67dda\") "
May 26 11:45:27.816908 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:45:27.816697    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/f71a756c-64ed-4d91-90d9-c27281c67dda-tokenizer-uds\") pod \"f71a756c-64ed-4d91-90d9-c27281c67dda\" (UID: \"f71a756c-64ed-4d91-90d9-c27281c67dda\") "
May 26 11:45:27.816908 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:45:27.816756    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/f71a756c-64ed-4d91-90d9-c27281c67dda-tokenizer-tmp\") pod \"f71a756c-64ed-4d91-90d9-c27281c67dda\" (UID: \"f71a756c-64ed-4d91-90d9-c27281c67dda\") "
May 26 11:45:27.816908 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:45:27.816823    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bm2rx\" (UniqueName: \"kubernetes.io/projected/f71a756c-64ed-4d91-90d9-c27281c67dda-kube-api-access-bm2rx\") pod \"f71a756c-64ed-4d91-90d9-c27281c67dda\" (UID: \"f71a756c-64ed-4d91-90d9-c27281c67dda\") "
May 26 11:45:27.816908 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:45:27.816850    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f71a756c-64ed-4d91-90d9-c27281c67dda-tls-certs\") pod \"f71a756c-64ed-4d91-90d9-c27281c67dda\" (UID: \"f71a756c-64ed-4d91-90d9-c27281c67dda\") "
May 26 11:45:27.817104 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:45:27.816931    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f71a756c-64ed-4d91-90d9-c27281c67dda-tokenizer-cache" (OuterVolumeSpecName: "tokenizer-cache") pod "f71a756c-64ed-4d91-90d9-c27281c67dda" (UID: "f71a756c-64ed-4d91-90d9-c27281c67dda"). InnerVolumeSpecName "tokenizer-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 11:45:27.817104 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:45:27.817046    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f71a756c-64ed-4d91-90d9-c27281c67dda-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "f71a756c-64ed-4d91-90d9-c27281c67dda" (UID: "f71a756c-64ed-4d91-90d9-c27281c67dda"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 11:45:27.817250 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:45:27.817133    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f71a756c-64ed-4d91-90d9-c27281c67dda-tokenizer-tmp" (OuterVolumeSpecName: "tokenizer-tmp") pod "f71a756c-64ed-4d91-90d9-c27281c67dda" (UID: "f71a756c-64ed-4d91-90d9-c27281c67dda"). InnerVolumeSpecName "tokenizer-tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 11:45:27.817250 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:45:27.817217    2572 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/f71a756c-64ed-4d91-90d9-c27281c67dda-tokenizer-uds\") on node \"ip-10-0-136-194.ec2.internal\" DevicePath \"\""
May 26 11:45:27.817250 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:45:27.817235    2572 reconciler_common.go:299] "Volume detached for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/f71a756c-64ed-4d91-90d9-c27281c67dda-tokenizer-cache\") on node \"ip-10-0-136-194.ec2.internal\" DevicePath \"\""
May 26 11:45:27.817570 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:45:27.817543    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f71a756c-64ed-4d91-90d9-c27281c67dda-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f71a756c-64ed-4d91-90d9-c27281c67dda" (UID: "f71a756c-64ed-4d91-90d9-c27281c67dda"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 11:45:27.819103 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:45:27.819075    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f71a756c-64ed-4d91-90d9-c27281c67dda-kube-api-access-bm2rx" (OuterVolumeSpecName: "kube-api-access-bm2rx") pod "f71a756c-64ed-4d91-90d9-c27281c67dda" (UID: "f71a756c-64ed-4d91-90d9-c27281c67dda"). InnerVolumeSpecName "kube-api-access-bm2rx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 11:45:27.819246 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:45:27.819125    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f71a756c-64ed-4d91-90d9-c27281c67dda-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f71a756c-64ed-4d91-90d9-c27281c67dda" (UID: "f71a756c-64ed-4d91-90d9-c27281c67dda"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 11:45:27.917888 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:45:27.917854    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bm2rx\" (UniqueName: \"kubernetes.io/projected/f71a756c-64ed-4d91-90d9-c27281c67dda-kube-api-access-bm2rx\") on node \"ip-10-0-136-194.ec2.internal\" DevicePath \"\""
May 26 11:45:27.917888 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:45:27.917885    2572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f71a756c-64ed-4d91-90d9-c27281c67dda-tls-certs\") on node \"ip-10-0-136-194.ec2.internal\" DevicePath \"\""
May 26 11:45:27.917888 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:45:27.917896    2572 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f71a756c-64ed-4d91-90d9-c27281c67dda-kserve-provision-location\") on node \"ip-10-0-136-194.ec2.internal\" DevicePath \"\""
May 26 11:45:27.918245 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:45:27.917915    2572 reconciler_common.go:299] "Volume detached for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/f71a756c-64ed-4d91-90d9-c27281c67dda-tokenizer-tmp\") on node \"ip-10-0-136-194.ec2.internal\" DevicePath \"\""
May 26 11:45:27.943707 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:45:27.943674    2572 generic.go:358] "Generic (PLEG): container finished" podID="031b53f5-7e1a-4a15-82d2-b6695d1f32eb" containerID="e7689fecc5b094f8ad74e467e750e31a9612c06ab731ba34fc49958d46d34703" exitCode=0
May 26 11:45:27.943890 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:45:27.943757    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-855b4f5774-tp6z7" event={"ID":"031b53f5-7e1a-4a15-82d2-b6695d1f32eb","Type":"ContainerDied","Data":"e7689fecc5b094f8ad74e467e750e31a9612c06ab731ba34fc49958d46d34703"}
May 26 11:45:27.943890 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:45:27.943791    2572 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-855b4f5774-tp6z7"
May 26 11:45:27.943890 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:45:27.943800    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-855b4f5774-tp6z7" event={"ID":"031b53f5-7e1a-4a15-82d2-b6695d1f32eb","Type":"ContainerDied","Data":"a3960eceaa2bee25df5937de96f17801b3840268e7c3e775428c35ae9123eec2"}
May 26 11:45:27.943890 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:45:27.943819    2572 scope.go:117] "RemoveContainer" containerID="e7689fecc5b094f8ad74e467e750e31a9612c06ab731ba34fc49958d46d34703"
May 26 11:45:27.945877 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:45:27.945859    2572 generic.go:358] "Generic (PLEG): container finished" podID="f71a756c-64ed-4d91-90d9-c27281c67dda" containerID="b4f452eadfdd49304c858d322e8ee0e1801878cb9003e4cec9b029d68ca7c2ad" exitCode=0
May 26 11:45:27.946003 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:45:27.945955    2572 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-747455452ls6"
May 26 11:45:27.946068 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:45:27.945949    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-747455452ls6" event={"ID":"f71a756c-64ed-4d91-90d9-c27281c67dda","Type":"ContainerDied","Data":"b4f452eadfdd49304c858d322e8ee0e1801878cb9003e4cec9b029d68ca7c2ad"}
May 26 11:45:27.946125 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:45:27.946090    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-747455452ls6" event={"ID":"f71a756c-64ed-4d91-90d9-c27281c67dda","Type":"ContainerDied","Data":"79099299b4b5ad7611dc5185ec9640e2912c3d032bb62fdc39caa25165952738"}
May 26 11:45:27.954017 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:45:27.953883    2572 scope.go:117] "RemoveContainer" containerID="8834b4a791cdd3db0ef2da1cbf852a858280f117525baa6fb788cabf4e9de6d3"
May 26 11:45:27.962254 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:45:27.962230    2572 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-855b4f5774-tp6z7"]
May 26 11:45:27.966405 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:45:27.966380    2572 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-855b4f5774-tp6z7"]
May 26 11:45:27.983583 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:45:27.983562    2572 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-747455452ls6"]
May 26 11:45:27.986621 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:45:27.986602    2572 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-747455452ls6"]
May 26 11:45:28.020208 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:45:28.020161    2572 scope.go:117] "RemoveContainer" containerID="e7689fecc5b094f8ad74e467e750e31a9612c06ab731ba34fc49958d46d34703"
May 26 11:45:28.020534 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:45:28.020506    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7689fecc5b094f8ad74e467e750e31a9612c06ab731ba34fc49958d46d34703\": container with ID starting with e7689fecc5b094f8ad74e467e750e31a9612c06ab731ba34fc49958d46d34703 not found: ID does not exist" containerID="e7689fecc5b094f8ad74e467e750e31a9612c06ab731ba34fc49958d46d34703"
May 26 11:45:28.020661 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:45:28.020542    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7689fecc5b094f8ad74e467e750e31a9612c06ab731ba34fc49958d46d34703"} err="failed to get container status \"e7689fecc5b094f8ad74e467e750e31a9612c06ab731ba34fc49958d46d34703\": rpc error: code = NotFound desc = could not find container \"e7689fecc5b094f8ad74e467e750e31a9612c06ab731ba34fc49958d46d34703\": container with ID starting with e7689fecc5b094f8ad74e467e750e31a9612c06ab731ba34fc49958d46d34703 not found: ID does not exist"
May 26 11:45:28.020661 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:45:28.020570    2572 scope.go:117] "RemoveContainer" containerID="8834b4a791cdd3db0ef2da1cbf852a858280f117525baa6fb788cabf4e9de6d3"
May 26 11:45:28.020864 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:45:28.020846    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8834b4a791cdd3db0ef2da1cbf852a858280f117525baa6fb788cabf4e9de6d3\": container with ID starting with 8834b4a791cdd3db0ef2da1cbf852a858280f117525baa6fb788cabf4e9de6d3 not found: ID does not exist" containerID="8834b4a791cdd3db0ef2da1cbf852a858280f117525baa6fb788cabf4e9de6d3"
May 26 11:45:28.020926 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:45:28.020875    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8834b4a791cdd3db0ef2da1cbf852a858280f117525baa6fb788cabf4e9de6d3"} err="failed to get container status \"8834b4a791cdd3db0ef2da1cbf852a858280f117525baa6fb788cabf4e9de6d3\": rpc error: code = NotFound desc = could not find container \"8834b4a791cdd3db0ef2da1cbf852a858280f117525baa6fb788cabf4e9de6d3\": container with ID starting with 8834b4a791cdd3db0ef2da1cbf852a858280f117525baa6fb788cabf4e9de6d3 not found: ID does not exist"
May 26 11:45:28.020926 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:45:28.020901    2572 scope.go:117] "RemoveContainer" containerID="b4f452eadfdd49304c858d322e8ee0e1801878cb9003e4cec9b029d68ca7c2ad"
May 26 11:45:28.029140 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:45:28.029120    2572 scope.go:117] "RemoveContainer" containerID="988498d845e1368772fe4eed4d7785ac80ffa9df95a44074af870456e95081b8"
May 26 11:45:28.036386 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:45:28.036372    2572 scope.go:117] "RemoveContainer" containerID="4a0d697bfb1b5cd406a987e9defccc1305ba3c7062ba03623e19fe3a610e653a"
May 26 11:45:28.043650 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:45:28.043635    2572 scope.go:117] "RemoveContainer" containerID="b4f452eadfdd49304c858d322e8ee0e1801878cb9003e4cec9b029d68ca7c2ad"
May 26 11:45:28.043881 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:45:28.043865    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4f452eadfdd49304c858d322e8ee0e1801878cb9003e4cec9b029d68ca7c2ad\": container with ID starting with b4f452eadfdd49304c858d322e8ee0e1801878cb9003e4cec9b029d68ca7c2ad not found: ID does not exist" containerID="b4f452eadfdd49304c858d322e8ee0e1801878cb9003e4cec9b029d68ca7c2ad"
May 26 11:45:28.043932 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:45:28.043887    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4f452eadfdd49304c858d322e8ee0e1801878cb9003e4cec9b029d68ca7c2ad"} err="failed to get container status \"b4f452eadfdd49304c858d322e8ee0e1801878cb9003e4cec9b029d68ca7c2ad\": rpc error: code = NotFound desc = could not find container \"b4f452eadfdd49304c858d322e8ee0e1801878cb9003e4cec9b029d68ca7c2ad\": container with ID starting with b4f452eadfdd49304c858d322e8ee0e1801878cb9003e4cec9b029d68ca7c2ad not found: ID does not exist"
May 26 11:45:28.043932 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:45:28.043904    2572 scope.go:117] "RemoveContainer" containerID="988498d845e1368772fe4eed4d7785ac80ffa9df95a44074af870456e95081b8"
May 26 11:45:28.044124 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:45:28.044106    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"988498d845e1368772fe4eed4d7785ac80ffa9df95a44074af870456e95081b8\": container with ID starting with 988498d845e1368772fe4eed4d7785ac80ffa9df95a44074af870456e95081b8 not found: ID does not exist" containerID="988498d845e1368772fe4eed4d7785ac80ffa9df95a44074af870456e95081b8"
May 26 11:45:28.044166 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:45:28.044127    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"988498d845e1368772fe4eed4d7785ac80ffa9df95a44074af870456e95081b8"} err="failed to get container status \"988498d845e1368772fe4eed4d7785ac80ffa9df95a44074af870456e95081b8\": rpc error: code = NotFound desc = could not find container \"988498d845e1368772fe4eed4d7785ac80ffa9df95a44074af870456e95081b8\": container with ID starting with 988498d845e1368772fe4eed4d7785ac80ffa9df95a44074af870456e95081b8 not found: ID does not exist"
May 26 11:45:28.044166 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:45:28.044140    2572 scope.go:117] "RemoveContainer" containerID="4a0d697bfb1b5cd406a987e9defccc1305ba3c7062ba03623e19fe3a610e653a"
May 26 11:45:28.044385 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:45:28.044365    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a0d697bfb1b5cd406a987e9defccc1305ba3c7062ba03623e19fe3a610e653a\": container with ID starting with 4a0d697bfb1b5cd406a987e9defccc1305ba3c7062ba03623e19fe3a610e653a not found: ID does not exist" containerID="4a0d697bfb1b5cd406a987e9defccc1305ba3c7062ba03623e19fe3a610e653a"
May 26 11:45:28.044473 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:45:28.044389    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a0d697bfb1b5cd406a987e9defccc1305ba3c7062ba03623e19fe3a610e653a"} err="failed to get container status \"4a0d697bfb1b5cd406a987e9defccc1305ba3c7062ba03623e19fe3a610e653a\": rpc error: code = NotFound desc = could not find container \"4a0d697bfb1b5cd406a987e9defccc1305ba3c7062ba03623e19fe3a610e653a\": container with ID starting with 4a0d697bfb1b5cd406a987e9defccc1305ba3c7062ba03623e19fe3a610e653a not found: ID does not exist"
May 26 11:45:29.677024 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:45:29.676989    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="031b53f5-7e1a-4a15-82d2-b6695d1f32eb" path="/var/lib/kubelet/pods/031b53f5-7e1a-4a15-82d2-b6695d1f32eb/volumes"
May 26 11:45:29.677477 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:45:29.677461    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f71a756c-64ed-4d91-90d9-c27281c67dda" path="/var/lib/kubelet/pods/f71a756c-64ed-4d91-90d9-c27281c67dda/volumes"
May 26 11:45:32.550324 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:45:32.550275    2572 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-b9b644d5d-54xv9" podUID="1ed85c58-ef71-4fde-b353-88d665ab6b39" containerName="main" probeResult="failure" output="Get \"https://10.132.0.37:8000/health\": dial tcp 10.132.0.37:8000: connect: connection refused"
May 26 11:45:33.631203 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:45:33.631157    2572 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-84f494576-mhtjs"]
May 26 11:45:33.631750 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:45:33.631728    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="031b53f5-7e1a-4a15-82d2-b6695d1f32eb" containerName="storage-initializer"
May 26 11:45:33.631832 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:45:33.631753    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="031b53f5-7e1a-4a15-82d2-b6695d1f32eb" containerName="storage-initializer"
May 26 11:45:33.631832 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:45:33.631765    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f71a756c-64ed-4d91-90d9-c27281c67dda" containerName="main"
May 26 11:45:33.631832 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:45:33.631774    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="f71a756c-64ed-4d91-90d9-c27281c67dda" containerName="main"
May 26 11:45:33.631832 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:45:33.631785    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f71a756c-64ed-4d91-90d9-c27281c67dda" containerName="tokenizer"
May 26 11:45:33.631832 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:45:33.631793    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="f71a756c-64ed-4d91-90d9-c27281c67dda" containerName="tokenizer"
May 26 11:45:33.631832 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:45:33.631820    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f71a756c-64ed-4d91-90d9-c27281c67dda" containerName="storage-initializer"
May 26 11:45:33.631832 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:45:33.631829    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="f71a756c-64ed-4d91-90d9-c27281c67dda" containerName="storage-initializer"
May 26 11:45:33.632225 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:45:33.631840    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="031b53f5-7e1a-4a15-82d2-b6695d1f32eb" containerName="main"
May 26 11:45:33.632225 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:45:33.631848    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="031b53f5-7e1a-4a15-82d2-b6695d1f32eb" containerName="main"
May 26 11:45:33.632225 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:45:33.631953    2572 memory_manager.go:356] "RemoveStaleState removing state" podUID="f71a756c-64ed-4d91-90d9-c27281c67dda" containerName="main"
May 26 11:45:33.632225 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:45:33.631970    2572 memory_manager.go:356] "RemoveStaleState removing state" podUID="031b53f5-7e1a-4a15-82d2-b6695d1f32eb" containerName="main"
May 26 11:45:33.632225 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:45:33.631983    2572 memory_manager.go:356] "RemoveStaleState removing state" podUID="f71a756c-64ed-4d91-90d9-c27281c67dda" containerName="tokenizer"
May 26 11:45:33.674831 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:45:33.674793    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-84f494576-mhtjs"]
May 26 11:45:33.674983 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:45:33.674938    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-84f494576-mhtjs"
May 26 11:45:33.677883 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:45:33.677854    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"precise-prefix-cache-test-kserve-self-signed-certs\""
May 26 11:45:33.770589 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:45:33.770551    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec94c1ea-a125-4069-a82d-9d5b6c2712ed-dshm\") pod \"precise-prefix-cache-test-kserve-84f494576-mhtjs\" (UID: \"ec94c1ea-a125-4069-a82d-9d5b6c2712ed\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-84f494576-mhtjs"
May 26 11:45:33.770589 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:45:33.770588    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec94c1ea-a125-4069-a82d-9d5b6c2712ed-tmp-dir\") pod \"precise-prefix-cache-test-kserve-84f494576-mhtjs\" (UID: \"ec94c1ea-a125-4069-a82d-9d5b6c2712ed\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-84f494576-mhtjs"
May 26 11:45:33.770798 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:45:33.770645    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ec94c1ea-a125-4069-a82d-9d5b6c2712ed-kserve-provision-location\") pod \"precise-prefix-cache-test-kserve-84f494576-mhtjs\" (UID: \"ec94c1ea-a125-4069-a82d-9d5b6c2712ed\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-84f494576-mhtjs"
May 26 11:45:33.770798 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:45:33.770713    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec94c1ea-a125-4069-a82d-9d5b6c2712ed-model-cache\") pod \"precise-prefix-cache-test-kserve-84f494576-mhtjs\" (UID: \"ec94c1ea-a125-4069-a82d-9d5b6c2712ed\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-84f494576-mhtjs"
May 26 11:45:33.770798 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:45:33.770752    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec94c1ea-a125-4069-a82d-9d5b6c2712ed-home\") pod \"precise-prefix-cache-test-kserve-84f494576-mhtjs\" (UID: \"ec94c1ea-a125-4069-a82d-9d5b6c2712ed\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-84f494576-mhtjs"
May 26 11:45:33.770798 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:45:33.770779    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec94c1ea-a125-4069-a82d-9d5b6c2712ed-tls-certs\") pod \"precise-prefix-cache-test-kserve-84f494576-mhtjs\" (UID: \"ec94c1ea-a125-4069-a82d-9d5b6c2712ed\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-84f494576-mhtjs"
May 26 11:45:33.770951 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:45:33.770866    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-646px\" (UniqueName: \"kubernetes.io/projected/ec94c1ea-a125-4069-a82d-9d5b6c2712ed-kube-api-access-646px\") pod \"precise-prefix-cache-test-kserve-84f494576-mhtjs\" (UID: \"ec94c1ea-a125-4069-a82d-9d5b6c2712ed\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-84f494576-mhtjs"
May 26 11:45:33.871736 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:45:33.871703    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-646px\" (UniqueName: \"kubernetes.io/projected/ec94c1ea-a125-4069-a82d-9d5b6c2712ed-kube-api-access-646px\") pod \"precise-prefix-cache-test-kserve-84f494576-mhtjs\" (UID: \"ec94c1ea-a125-4069-a82d-9d5b6c2712ed\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-84f494576-mhtjs"
May 26 11:45:33.871932 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:45:33.871763    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec94c1ea-a125-4069-a82d-9d5b6c2712ed-dshm\") pod \"precise-prefix-cache-test-kserve-84f494576-mhtjs\" (UID: \"ec94c1ea-a125-4069-a82d-9d5b6c2712ed\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-84f494576-mhtjs"
May 26 11:45:33.871932 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:45:33.871792    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec94c1ea-a125-4069-a82d-9d5b6c2712ed-tmp-dir\") pod \"precise-prefix-cache-test-kserve-84f494576-mhtjs\" (UID: \"ec94c1ea-a125-4069-a82d-9d5b6c2712ed\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-84f494576-mhtjs"
May 26 11:45:33.871932 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:45:33.871835    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ec94c1ea-a125-4069-a82d-9d5b6c2712ed-kserve-provision-location\") pod \"precise-prefix-cache-test-kserve-84f494576-mhtjs\" (UID: \"ec94c1ea-a125-4069-a82d-9d5b6c2712ed\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-84f494576-mhtjs"
May 26 11:45:33.871932 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:45:33.871901    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec94c1ea-a125-4069-a82d-9d5b6c2712ed-model-cache\") pod \"precise-prefix-cache-test-kserve-84f494576-mhtjs\" (UID: \"ec94c1ea-a125-4069-a82d-9d5b6c2712ed\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-84f494576-mhtjs"
May 26 11:45:33.872129 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:45:33.872076    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec94c1ea-a125-4069-a82d-9d5b6c2712ed-home\") pod \"precise-prefix-cache-test-kserve-84f494576-mhtjs\" (UID: \"ec94c1ea-a125-4069-a82d-9d5b6c2712ed\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-84f494576-mhtjs"
May 26 11:45:33.872182 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:45:33.872142    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec94c1ea-a125-4069-a82d-9d5b6c2712ed-tls-certs\") pod \"precise-prefix-cache-test-kserve-84f494576-mhtjs\" (UID: \"ec94c1ea-a125-4069-a82d-9d5b6c2712ed\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-84f494576-mhtjs"
May 26 11:45:33.872266 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:45:33.872244    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec94c1ea-a125-4069-a82d-9d5b6c2712ed-tmp-dir\") pod \"precise-prefix-cache-test-kserve-84f494576-mhtjs\" (UID: \"ec94c1ea-a125-4069-a82d-9d5b6c2712ed\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-84f494576-mhtjs"
May 26 11:45:33.872327 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:45:33.872278    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ec94c1ea-a125-4069-a82d-9d5b6c2712ed-kserve-provision-location\") pod \"precise-prefix-cache-test-kserve-84f494576-mhtjs\" (UID: \"ec94c1ea-a125-4069-a82d-9d5b6c2712ed\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-84f494576-mhtjs"
May 26 11:45:33.872401 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:45:33.872382    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec94c1ea-a125-4069-a82d-9d5b6c2712ed-model-cache\") pod \"precise-prefix-cache-test-kserve-84f494576-mhtjs\" (UID: \"ec94c1ea-a125-4069-a82d-9d5b6c2712ed\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-84f494576-mhtjs"
May 26 11:45:33.872518 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:45:33.872494    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec94c1ea-a125-4069-a82d-9d5b6c2712ed-home\") pod \"precise-prefix-cache-test-kserve-84f494576-mhtjs\" (UID: \"ec94c1ea-a125-4069-a82d-9d5b6c2712ed\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-84f494576-mhtjs"
May 26 11:45:33.874301 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:45:33.874273    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec94c1ea-a125-4069-a82d-9d5b6c2712ed-dshm\") pod \"precise-prefix-cache-test-kserve-84f494576-mhtjs\" (UID: \"ec94c1ea-a125-4069-a82d-9d5b6c2712ed\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-84f494576-mhtjs"
May 26 11:45:33.874778 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:45:33.874757    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec94c1ea-a125-4069-a82d-9d5b6c2712ed-tls-certs\") pod \"precise-prefix-cache-test-kserve-84f494576-mhtjs\" (UID: \"ec94c1ea-a125-4069-a82d-9d5b6c2712ed\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-84f494576-mhtjs"
May 26 11:45:33.882017 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:45:33.881937    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-646px\" (UniqueName: \"kubernetes.io/projected/ec94c1ea-a125-4069-a82d-9d5b6c2712ed-kube-api-access-646px\") pod \"precise-prefix-cache-test-kserve-84f494576-mhtjs\" (UID: \"ec94c1ea-a125-4069-a82d-9d5b6c2712ed\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-84f494576-mhtjs"
May 26 11:45:33.907085 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:45:33.907054    2572 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-858b6cd9j49rn"]
May 26 11:45:33.940876 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:45:33.940845    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-858b6cd9j49rn"]
May 26 11:45:33.941059 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:45:33.941000    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-858b6cd9j49rn"
May 26 11:45:33.944216 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:45:33.944180    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"precise-prefix-cache-test-epp-sa-dockercfg-9m8hw\""
May 26 11:45:33.973641 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:45:33.973604    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2039ec6-c16e-495f-927a-97103f09da12-tls-certs\") pod \"precise-prefix-cache-test-kserve-router-scheduler-858b6cd9j49rn\" (UID: \"d2039ec6-c16e-495f-927a-97103f09da12\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-858b6cd9j49rn"
May 26 11:45:33.973815 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:45:33.973676    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/d2039ec6-c16e-495f-927a-97103f09da12-tokenizer-uds\") pod \"precise-prefix-cache-test-kserve-router-scheduler-858b6cd9j49rn\" (UID: \"d2039ec6-c16e-495f-927a-97103f09da12\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-858b6cd9j49rn"
May 26 11:45:33.973815 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:45:33.973724    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d2039ec6-c16e-495f-927a-97103f09da12-kserve-provision-location\") pod \"precise-prefix-cache-test-kserve-router-scheduler-858b6cd9j49rn\" (UID: \"d2039ec6-c16e-495f-927a-97103f09da12\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-858b6cd9j49rn"
May 26 11:45:33.973947 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:45:33.973836    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cw6hw\" (UniqueName: \"kubernetes.io/projected/d2039ec6-c16e-495f-927a-97103f09da12-kube-api-access-cw6hw\") pod \"precise-prefix-cache-test-kserve-router-scheduler-858b6cd9j49rn\" (UID: \"d2039ec6-c16e-495f-927a-97103f09da12\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-858b6cd9j49rn"
May 26 11:45:33.973947 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:45:33.973866    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2039ec6-c16e-495f-927a-97103f09da12-tokenizer-cache\") pod \"precise-prefix-cache-test-kserve-router-scheduler-858b6cd9j49rn\" (UID: \"d2039ec6-c16e-495f-927a-97103f09da12\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-858b6cd9j49rn"
May 26 11:45:33.973947 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:45:33.973898    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/d2039ec6-c16e-495f-927a-97103f09da12-tokenizer-tmp\") pod \"precise-prefix-cache-test-kserve-router-scheduler-858b6cd9j49rn\" (UID: \"d2039ec6-c16e-495f-927a-97103f09da12\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-858b6cd9j49rn"
May 26 11:45:33.985649 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:45:33.985616    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-84f494576-mhtjs"
May 26 11:45:34.074904 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:45:34.074856    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/d2039ec6-c16e-495f-927a-97103f09da12-tokenizer-uds\") pod \"precise-prefix-cache-test-kserve-router-scheduler-858b6cd9j49rn\" (UID: \"d2039ec6-c16e-495f-927a-97103f09da12\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-858b6cd9j49rn"
May 26 11:45:34.075053 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:45:34.074909    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d2039ec6-c16e-495f-927a-97103f09da12-kserve-provision-location\") pod \"precise-prefix-cache-test-kserve-router-scheduler-858b6cd9j49rn\" (UID: \"d2039ec6-c16e-495f-927a-97103f09da12\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-858b6cd9j49rn"
May 26 11:45:34.075053 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:45:34.074992    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-cw6hw\" (UniqueName: \"kubernetes.io/projected/d2039ec6-c16e-495f-927a-97103f09da12-kube-api-access-cw6hw\") pod \"precise-prefix-cache-test-kserve-router-scheduler-858b6cd9j49rn\" (UID: \"d2039ec6-c16e-495f-927a-97103f09da12\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-858b6cd9j49rn"
May 26 11:45:34.075053 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:45:34.075019    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2039ec6-c16e-495f-927a-97103f09da12-tokenizer-cache\") pod \"precise-prefix-cache-test-kserve-router-scheduler-858b6cd9j49rn\" (UID: \"d2039ec6-c16e-495f-927a-97103f09da12\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-858b6cd9j49rn"
May 26 11:45:34.075265 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:45:34.075062    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/d2039ec6-c16e-495f-927a-97103f09da12-tokenizer-tmp\") pod \"precise-prefix-cache-test-kserve-router-scheduler-858b6cd9j49rn\" (UID: \"d2039ec6-c16e-495f-927a-97103f09da12\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-858b6cd9j49rn"
May 26 11:45:34.075265 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:45:34.075108    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2039ec6-c16e-495f-927a-97103f09da12-tls-certs\") pod \"precise-prefix-cache-test-kserve-router-scheduler-858b6cd9j49rn\" (UID: \"d2039ec6-c16e-495f-927a-97103f09da12\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-858b6cd9j49rn"
May 26 11:45:34.075388 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:45:34.075361    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/d2039ec6-c16e-495f-927a-97103f09da12-tokenizer-uds\") pod \"precise-prefix-cache-test-kserve-router-scheduler-858b6cd9j49rn\" (UID: \"d2039ec6-c16e-495f-927a-97103f09da12\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-858b6cd9j49rn"
May 26 11:45:34.075695 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:45:34.075611    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2039ec6-c16e-495f-927a-97103f09da12-tokenizer-cache\") pod \"precise-prefix-cache-test-kserve-router-scheduler-858b6cd9j49rn\" (UID: \"d2039ec6-c16e-495f-927a-97103f09da12\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-858b6cd9j49rn"
May 26 11:45:34.075695 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:45:34.075647    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d2039ec6-c16e-495f-927a-97103f09da12-kserve-provision-location\") pod \"precise-prefix-cache-test-kserve-router-scheduler-858b6cd9j49rn\" (UID: \"d2039ec6-c16e-495f-927a-97103f09da12\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-858b6cd9j49rn"
May 26 11:45:34.075695 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:45:34.075658    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/d2039ec6-c16e-495f-927a-97103f09da12-tokenizer-tmp\") pod \"precise-prefix-cache-test-kserve-router-scheduler-858b6cd9j49rn\" (UID: \"d2039ec6-c16e-495f-927a-97103f09da12\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-858b6cd9j49rn"
May 26 11:45:34.078241 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:45:34.078217    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2039ec6-c16e-495f-927a-97103f09da12-tls-certs\") pod \"precise-prefix-cache-test-kserve-router-scheduler-858b6cd9j49rn\" (UID: \"d2039ec6-c16e-495f-927a-97103f09da12\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-858b6cd9j49rn"
May 26 11:45:34.083360 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:45:34.083337    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cw6hw\" (UniqueName: \"kubernetes.io/projected/d2039ec6-c16e-495f-927a-97103f09da12-kube-api-access-cw6hw\") pod \"precise-prefix-cache-test-kserve-router-scheduler-858b6cd9j49rn\" (UID: \"d2039ec6-c16e-495f-927a-97103f09da12\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-858b6cd9j49rn"
May 26 11:45:34.126769 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:45:34.126735    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-84f494576-mhtjs"]
May 26 11:45:34.127285 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:45:34.127257    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec94c1ea_a125_4069_a82d_9d5b6c2712ed.slice/crio-8a21836305165d99b3c1007cbd57e61358c23077179812149d2dc8c730316cd1 WatchSource:0}: Error finding container 8a21836305165d99b3c1007cbd57e61358c23077179812149d2dc8c730316cd1: Status 404 returned error can't find the container with id 8a21836305165d99b3c1007cbd57e61358c23077179812149d2dc8c730316cd1
May 26 11:45:34.129208 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:45:34.129167    2572 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 26 11:45:34.252043 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:45:34.252016    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-858b6cd9j49rn"
May 26 11:45:34.393633 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:45:34.393603    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-858b6cd9j49rn"]
May 26 11:45:34.395854 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:45:34.395825    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2039ec6_c16e_495f_927a_97103f09da12.slice/crio-8f550c1ddd29128ec4a97805299bee6e444e8745ef838296da0c2283117ebe06 WatchSource:0}: Error finding container 8f550c1ddd29128ec4a97805299bee6e444e8745ef838296da0c2283117ebe06: Status 404 returned error can't find the container with id 8f550c1ddd29128ec4a97805299bee6e444e8745ef838296da0c2283117ebe06
May 26 11:45:34.974000 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:45:34.973958    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-858b6cd9j49rn" event={"ID":"d2039ec6-c16e-495f-927a-97103f09da12","Type":"ContainerStarted","Data":"caad9e3b463dc44220995a7fe867859fd8372fe695054f9b2c10f08a4d383004"}
May 26 11:45:34.974000 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:45:34.974004    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-858b6cd9j49rn" event={"ID":"d2039ec6-c16e-495f-927a-97103f09da12","Type":"ContainerStarted","Data":"8f550c1ddd29128ec4a97805299bee6e444e8745ef838296da0c2283117ebe06"}
May 26 11:45:34.975513 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:45:34.975485    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-84f494576-mhtjs" event={"ID":"ec94c1ea-a125-4069-a82d-9d5b6c2712ed","Type":"ContainerStarted","Data":"03bbd6d099de04026f99a1ccccbfe36429f0df9b00c7f3eabc1b149b45e1ba56"}
May 26 11:45:34.975652 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:45:34.975517    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-84f494576-mhtjs" event={"ID":"ec94c1ea-a125-4069-a82d-9d5b6c2712ed","Type":"ContainerStarted","Data":"8a21836305165d99b3c1007cbd57e61358c23077179812149d2dc8c730316cd1"}
May 26 11:45:35.980697 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:45:35.980654    2572 generic.go:358] "Generic (PLEG): container finished" podID="d2039ec6-c16e-495f-927a-97103f09da12" containerID="caad9e3b463dc44220995a7fe867859fd8372fe695054f9b2c10f08a4d383004" exitCode=0
May 26 11:45:35.981301 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:45:35.980753    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-858b6cd9j49rn" event={"ID":"d2039ec6-c16e-495f-927a-97103f09da12","Type":"ContainerDied","Data":"caad9e3b463dc44220995a7fe867859fd8372fe695054f9b2c10f08a4d383004"}
May 26 11:45:36.987715 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:45:36.987682    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-858b6cd9j49rn" event={"ID":"d2039ec6-c16e-495f-927a-97103f09da12","Type":"ContainerStarted","Data":"c3afc764fe2c252960e9df246eb7da69c01deefeb8f4d916a217d08dc8fd5b3a"}
May 26 11:45:36.987715 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:45:36.987720    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-858b6cd9j49rn" event={"ID":"d2039ec6-c16e-495f-927a-97103f09da12","Type":"ContainerStarted","Data":"86721cc462c35ec1864092fa3b76d19334298dff17dc41e001a610ca161ad6b0"}
May 26 11:45:36.988150 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:45:36.987859    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-858b6cd9j49rn"
May 26 11:45:37.008658 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:45:37.008610    2572 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-858b6cd9j49rn" podStartSLOduration=4.008594794 podStartE2EDuration="4.008594794s" podCreationTimestamp="2026-05-26 11:45:33 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 11:45:37.006128387 +0000 UTC m=+1259.908338719" watchObservedRunningTime="2026-05-26 11:45:37.008594794 +0000 UTC m=+1259.910805123"
May 26 11:45:42.550460 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:45:42.550416    2572 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-b9b644d5d-54xv9" podUID="1ed85c58-ef71-4fde-b353-88d665ab6b39" containerName="main" probeResult="failure" output="Get \"https://10.132.0.37:8000/health\": dial tcp 10.132.0.37:8000: connect: connection refused"
May 26 11:45:44.253142 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:45:44.253100    2572 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-858b6cd9j49rn"
May 26 11:45:44.253602 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:45:44.253162    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-858b6cd9j49rn"
May 26 11:45:44.254461 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:45:44.254429    2572 logging.go:55] [core] [Channel #62 SubChannel #63]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.42:9003", ServerName: "10.132.0.42:9003", }. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.42:9003: connect: connection refused"
May 26 11:45:44.255759 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:45:44.255737    2572 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-858b6cd9j49rn"
May 26 11:45:45.019167 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:45:45.019138    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-858b6cd9j49rn"
May 26 11:45:45.253302 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:45:45.253258    2572 prober.go:120] "Probe failed" probeType="Liveness" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-858b6cd9j49rn" podUID="d2039ec6-c16e-495f-927a-97103f09da12" containerName="main" probeResult="failure" output="timeout: failed to connect service \"10.132.0.42:9003\" within 1s: context deadline exceeded"
May 26 11:45:52.550719 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:45:52.550670    2572 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-b9b644d5d-54xv9" podUID="1ed85c58-ef71-4fde-b353-88d665ab6b39" containerName="main" probeResult="failure" output="Get \"https://10.132.0.37:8000/health\": dial tcp 10.132.0.37:8000: connect: connection refused"
May 26 11:45:54.253180 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:45:54.253146    2572 logging.go:55] [core] [Channel #64 SubChannel #65]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.42:9003", ServerName: "10.132.0.42:9003", }. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.42:9003: connect: connection refused"
May 26 11:45:55.253374 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:45:55.253328    2572 prober.go:120] "Probe failed" probeType="Liveness" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-858b6cd9j49rn" podUID="d2039ec6-c16e-495f-927a-97103f09da12" containerName="main" probeResult="failure" output="timeout: failed to connect service \"10.132.0.42:9003\" within 1s: context deadline exceeded"
May 26 11:46:02.550063 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:46:02.550018    2572 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-b9b644d5d-54xv9" podUID="1ed85c58-ef71-4fde-b353-88d665ab6b39" containerName="main" probeResult="failure" output="Get \"https://10.132.0.37:8000/health\": dial tcp 10.132.0.37:8000: connect: connection refused"
May 26 11:46:06.022799 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:46:06.022768    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-858b6cd9j49rn"
May 26 11:46:12.549763 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:46:12.549722    2572 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-b9b644d5d-54xv9" podUID="1ed85c58-ef71-4fde-b353-88d665ab6b39" containerName="main" probeResult="failure" output="Get \"https://10.132.0.37:8000/health\": dial tcp 10.132.0.37:8000: connect: connection refused"
May 26 11:46:22.550495 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:46:22.550451    2572 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-b9b644d5d-54xv9" podUID="1ed85c58-ef71-4fde-b353-88d665ab6b39" containerName="main" probeResult="failure" output="Get \"https://10.132.0.37:8000/health\": dial tcp 10.132.0.37:8000: connect: connection refused"
May 26 11:46:32.550452 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:46:32.550406    2572 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-b9b644d5d-54xv9" podUID="1ed85c58-ef71-4fde-b353-88d665ab6b39" containerName="main" probeResult="failure" output="Get \"https://10.132.0.37:8000/health\": dial tcp 10.132.0.37:8000: connect: connection refused"
May 26 11:46:42.550485 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:46:42.550431    2572 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-b9b644d5d-54xv9" podUID="1ed85c58-ef71-4fde-b353-88d665ab6b39" containerName="main" probeResult="failure" output="Get \"https://10.132.0.37:8000/health\": dial tcp 10.132.0.37:8000: connect: connection refused"
May 26 11:46:52.559817 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:46:52.559780    2572 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-b9b644d5d-54xv9"
May 26 11:46:52.567520 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:46:52.567501    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-b9b644d5d-54xv9"
May 26 11:46:58.525715 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:46:58.525678    2572 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-b9b644d5d-54xv9"]
May 26 11:46:58.526096 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:46:58.526047    2572 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-b9b644d5d-54xv9" podUID="1ed85c58-ef71-4fde-b353-88d665ab6b39" containerName="main" containerID="cri-o://18c6e788cac9ba701a92001b7bf4cd9a11b0ff7bd1d9849a4277a49a93316783" gracePeriod=30
May 26 11:47:12.032730 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:47:12.032692    2572 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-65d8bf8b78-5cdq2"]
May 26 11:47:12.035321 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:47:12.035301    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-65d8bf8b78-5cdq2"
May 26 11:47:12.039181 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:47:12.039156    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"custom-route-timeout-test-kserve-self-signed-certs\""
May 26 11:47:12.051801 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:47:12.051773    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-65d8bf8b78-5cdq2"]
May 26 11:47:12.114260 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:47:12.114235    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nb4jn\" (UniqueName: \"kubernetes.io/projected/2048d8b5-e1bc-41b0-bd27-8a8228cea5c5-kube-api-access-nb4jn\") pod \"custom-route-timeout-test-kserve-65d8bf8b78-5cdq2\" (UID: \"2048d8b5-e1bc-41b0-bd27-8a8228cea5c5\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-65d8bf8b78-5cdq2"
May 26 11:47:12.114412 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:47:12.114282    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2048d8b5-e1bc-41b0-bd27-8a8228cea5c5-tmp-dir\") pod \"custom-route-timeout-test-kserve-65d8bf8b78-5cdq2\" (UID: \"2048d8b5-e1bc-41b0-bd27-8a8228cea5c5\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-65d8bf8b78-5cdq2"
May 26 11:47:12.114412 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:47:12.114308    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2048d8b5-e1bc-41b0-bd27-8a8228cea5c5-tls-certs\") pod \"custom-route-timeout-test-kserve-65d8bf8b78-5cdq2\" (UID: \"2048d8b5-e1bc-41b0-bd27-8a8228cea5c5\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-65d8bf8b78-5cdq2"
May 26 11:47:12.114412 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:47:12.114362    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2048d8b5-e1bc-41b0-bd27-8a8228cea5c5-home\") pod \"custom-route-timeout-test-kserve-65d8bf8b78-5cdq2\" (UID: \"2048d8b5-e1bc-41b0-bd27-8a8228cea5c5\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-65d8bf8b78-5cdq2"
May 26 11:47:12.114412 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:47:12.114391    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2048d8b5-e1bc-41b0-bd27-8a8228cea5c5-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-65d8bf8b78-5cdq2\" (UID: \"2048d8b5-e1bc-41b0-bd27-8a8228cea5c5\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-65d8bf8b78-5cdq2"
May 26 11:47:12.114616 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:47:12.114427    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2048d8b5-e1bc-41b0-bd27-8a8228cea5c5-dshm\") pod \"custom-route-timeout-test-kserve-65d8bf8b78-5cdq2\" (UID: \"2048d8b5-e1bc-41b0-bd27-8a8228cea5c5\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-65d8bf8b78-5cdq2"
May 26 11:47:12.114616 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:47:12.114519    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2048d8b5-e1bc-41b0-bd27-8a8228cea5c5-model-cache\") pod \"custom-route-timeout-test-kserve-65d8bf8b78-5cdq2\" (UID: \"2048d8b5-e1bc-41b0-bd27-8a8228cea5c5\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-65d8bf8b78-5cdq2"
May 26 11:47:12.215523 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:47:12.215494    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2048d8b5-e1bc-41b0-bd27-8a8228cea5c5-tmp-dir\") pod \"custom-route-timeout-test-kserve-65d8bf8b78-5cdq2\" (UID: \"2048d8b5-e1bc-41b0-bd27-8a8228cea5c5\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-65d8bf8b78-5cdq2"
May 26 11:47:12.215640 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:47:12.215527    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2048d8b5-e1bc-41b0-bd27-8a8228cea5c5-tls-certs\") pod \"custom-route-timeout-test-kserve-65d8bf8b78-5cdq2\" (UID: \"2048d8b5-e1bc-41b0-bd27-8a8228cea5c5\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-65d8bf8b78-5cdq2"
May 26 11:47:12.215640 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:47:12.215552    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2048d8b5-e1bc-41b0-bd27-8a8228cea5c5-home\") pod \"custom-route-timeout-test-kserve-65d8bf8b78-5cdq2\" (UID: \"2048d8b5-e1bc-41b0-bd27-8a8228cea5c5\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-65d8bf8b78-5cdq2"
May 26 11:47:12.215716 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:47:12.215673    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2048d8b5-e1bc-41b0-bd27-8a8228cea5c5-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-65d8bf8b78-5cdq2\" (UID: \"2048d8b5-e1bc-41b0-bd27-8a8228cea5c5\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-65d8bf8b78-5cdq2"
May 26 11:47:12.215716 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:47:12.215704    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2048d8b5-e1bc-41b0-bd27-8a8228cea5c5-dshm\") pod \"custom-route-timeout-test-kserve-65d8bf8b78-5cdq2\" (UID: \"2048d8b5-e1bc-41b0-bd27-8a8228cea5c5\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-65d8bf8b78-5cdq2"
May 26 11:47:12.215832 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:47:12.215814    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2048d8b5-e1bc-41b0-bd27-8a8228cea5c5-model-cache\") pod \"custom-route-timeout-test-kserve-65d8bf8b78-5cdq2\" (UID: \"2048d8b5-e1bc-41b0-bd27-8a8228cea5c5\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-65d8bf8b78-5cdq2"
May 26 11:47:12.215887 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:47:12.215856    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-nb4jn\" (UniqueName: \"kubernetes.io/projected/2048d8b5-e1bc-41b0-bd27-8a8228cea5c5-kube-api-access-nb4jn\") pod \"custom-route-timeout-test-kserve-65d8bf8b78-5cdq2\" (UID: \"2048d8b5-e1bc-41b0-bd27-8a8228cea5c5\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-65d8bf8b78-5cdq2"
May 26 11:47:12.215887 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:47:12.215867    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2048d8b5-e1bc-41b0-bd27-8a8228cea5c5-tmp-dir\") pod \"custom-route-timeout-test-kserve-65d8bf8b78-5cdq2\" (UID: \"2048d8b5-e1bc-41b0-bd27-8a8228cea5c5\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-65d8bf8b78-5cdq2"
May 26 11:47:12.215993 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:47:12.215914    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2048d8b5-e1bc-41b0-bd27-8a8228cea5c5-home\") pod \"custom-route-timeout-test-kserve-65d8bf8b78-5cdq2\" (UID: \"2048d8b5-e1bc-41b0-bd27-8a8228cea5c5\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-65d8bf8b78-5cdq2"
May 26 11:47:12.216051 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:47:12.216033    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2048d8b5-e1bc-41b0-bd27-8a8228cea5c5-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-65d8bf8b78-5cdq2\" (UID: \"2048d8b5-e1bc-41b0-bd27-8a8228cea5c5\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-65d8bf8b78-5cdq2"
May 26 11:47:12.216158 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:47:12.216136    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2048d8b5-e1bc-41b0-bd27-8a8228cea5c5-model-cache\") pod \"custom-route-timeout-test-kserve-65d8bf8b78-5cdq2\" (UID: \"2048d8b5-e1bc-41b0-bd27-8a8228cea5c5\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-65d8bf8b78-5cdq2"
May 26 11:47:12.217823 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:47:12.217804    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2048d8b5-e1bc-41b0-bd27-8a8228cea5c5-dshm\") pod \"custom-route-timeout-test-kserve-65d8bf8b78-5cdq2\" (UID: \"2048d8b5-e1bc-41b0-bd27-8a8228cea5c5\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-65d8bf8b78-5cdq2"
May 26 11:47:12.218015 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:47:12.218000    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2048d8b5-e1bc-41b0-bd27-8a8228cea5c5-tls-certs\") pod \"custom-route-timeout-test-kserve-65d8bf8b78-5cdq2\" (UID: \"2048d8b5-e1bc-41b0-bd27-8a8228cea5c5\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-65d8bf8b78-5cdq2"
May 26 11:47:12.228724 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:47:12.228700    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nb4jn\" (UniqueName: \"kubernetes.io/projected/2048d8b5-e1bc-41b0-bd27-8a8228cea5c5-kube-api-access-nb4jn\") pod \"custom-route-timeout-test-kserve-65d8bf8b78-5cdq2\" (UID: \"2048d8b5-e1bc-41b0-bd27-8a8228cea5c5\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-65d8bf8b78-5cdq2"
May 26 11:47:12.346311 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:47:12.346238    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-65d8bf8b78-5cdq2"
May 26 11:47:12.486859 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:47:12.486829    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-65d8bf8b78-5cdq2"]
May 26 11:47:12.488751 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:47:12.488723    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2048d8b5_e1bc_41b0_bd27_8a8228cea5c5.slice/crio-57825cdfc31aee39de0a7b6c28cc8fcb8bef124ca5b897f7efc4da9a9ae44069 WatchSource:0}: Error finding container 57825cdfc31aee39de0a7b6c28cc8fcb8bef124ca5b897f7efc4da9a9ae44069: Status 404 returned error can't find the container with id 57825cdfc31aee39de0a7b6c28cc8fcb8bef124ca5b897f7efc4da9a9ae44069
May 26 11:47:13.350383 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:47:13.350326    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-65d8bf8b78-5cdq2" event={"ID":"2048d8b5-e1bc-41b0-bd27-8a8228cea5c5","Type":"ContainerStarted","Data":"b75ac10c88206a9c640d9656f13e05e5e8017429b34b55c805d21844a4e0ed36"}
May 26 11:47:13.350383 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:47:13.350385    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-65d8bf8b78-5cdq2" event={"ID":"2048d8b5-e1bc-41b0-bd27-8a8228cea5c5","Type":"ContainerStarted","Data":"57825cdfc31aee39de0a7b6c28cc8fcb8bef124ca5b897f7efc4da9a9ae44069"}
May 26 11:47:28.773808 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:47:28.773784    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-b9b644d5d-54xv9_1ed85c58-ef71-4fde-b353-88d665ab6b39/main/0.log"
May 26 11:47:28.774172 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:47:28.774151    2572 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-b9b644d5d-54xv9"
May 26 11:47:28.874348 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:47:28.874277    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1ed85c58-ef71-4fde-b353-88d665ab6b39-kserve-provision-location\") pod \"1ed85c58-ef71-4fde-b353-88d665ab6b39\" (UID: \"1ed85c58-ef71-4fde-b353-88d665ab6b39\") "
May 26 11:47:28.874348 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:47:28.874339    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ed85c58-ef71-4fde-b353-88d665ab6b39-dshm\") pod \"1ed85c58-ef71-4fde-b353-88d665ab6b39\" (UID: \"1ed85c58-ef71-4fde-b353-88d665ab6b39\") "
May 26 11:47:28.874547 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:47:28.874357    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wvvsg\" (UniqueName: \"kubernetes.io/projected/1ed85c58-ef71-4fde-b353-88d665ab6b39-kube-api-access-wvvsg\") pod \"1ed85c58-ef71-4fde-b353-88d665ab6b39\" (UID: \"1ed85c58-ef71-4fde-b353-88d665ab6b39\") "
May 26 11:47:28.874547 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:47:28.874391    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ed85c58-ef71-4fde-b353-88d665ab6b39-model-cache\") pod \"1ed85c58-ef71-4fde-b353-88d665ab6b39\" (UID: \"1ed85c58-ef71-4fde-b353-88d665ab6b39\") "
May 26 11:47:28.874547 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:47:28.874415    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ed85c58-ef71-4fde-b353-88d665ab6b39-home\") pod \"1ed85c58-ef71-4fde-b353-88d665ab6b39\" (UID: \"1ed85c58-ef71-4fde-b353-88d665ab6b39\") "
May 26 11:47:28.874547 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:47:28.874461    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ed85c58-ef71-4fde-b353-88d665ab6b39-tmp-dir\") pod \"1ed85c58-ef71-4fde-b353-88d665ab6b39\" (UID: \"1ed85c58-ef71-4fde-b353-88d665ab6b39\") "
May 26 11:47:28.874547 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:47:28.874480    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ed85c58-ef71-4fde-b353-88d665ab6b39-tls-certs\") pod \"1ed85c58-ef71-4fde-b353-88d665ab6b39\" (UID: \"1ed85c58-ef71-4fde-b353-88d665ab6b39\") "
May 26 11:47:28.874800 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:47:28.874736    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ed85c58-ef71-4fde-b353-88d665ab6b39-model-cache" (OuterVolumeSpecName: "model-cache") pod "1ed85c58-ef71-4fde-b353-88d665ab6b39" (UID: "1ed85c58-ef71-4fde-b353-88d665ab6b39"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 11:47:28.875149 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:47:28.875120    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ed85c58-ef71-4fde-b353-88d665ab6b39-home" (OuterVolumeSpecName: "home") pod "1ed85c58-ef71-4fde-b353-88d665ab6b39" (UID: "1ed85c58-ef71-4fde-b353-88d665ab6b39"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 11:47:28.875149 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:47:28.875135    2572 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ed85c58-ef71-4fde-b353-88d665ab6b39-model-cache\") on node \"ip-10-0-136-194.ec2.internal\" DevicePath \"\""
May 26 11:47:28.876806 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:47:28.876785    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ed85c58-ef71-4fde-b353-88d665ab6b39-dshm" (OuterVolumeSpecName: "dshm") pod "1ed85c58-ef71-4fde-b353-88d665ab6b39" (UID: "1ed85c58-ef71-4fde-b353-88d665ab6b39"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 11:47:28.877051 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:47:28.877027    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ed85c58-ef71-4fde-b353-88d665ab6b39-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1ed85c58-ef71-4fde-b353-88d665ab6b39" (UID: "1ed85c58-ef71-4fde-b353-88d665ab6b39"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 11:47:28.877051 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:47:28.877040    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ed85c58-ef71-4fde-b353-88d665ab6b39-kube-api-access-wvvsg" (OuterVolumeSpecName: "kube-api-access-wvvsg") pod "1ed85c58-ef71-4fde-b353-88d665ab6b39" (UID: "1ed85c58-ef71-4fde-b353-88d665ab6b39"). InnerVolumeSpecName "kube-api-access-wvvsg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 11:47:28.891969 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:47:28.891939    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ed85c58-ef71-4fde-b353-88d665ab6b39-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1ed85c58-ef71-4fde-b353-88d665ab6b39" (UID: "1ed85c58-ef71-4fde-b353-88d665ab6b39"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 11:47:28.932020 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:47:28.931988    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ed85c58-ef71-4fde-b353-88d665ab6b39-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "1ed85c58-ef71-4fde-b353-88d665ab6b39" (UID: "1ed85c58-ef71-4fde-b353-88d665ab6b39"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 11:47:28.975844 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:47:28.975822    2572 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1ed85c58-ef71-4fde-b353-88d665ab6b39-kserve-provision-location\") on node \"ip-10-0-136-194.ec2.internal\" DevicePath \"\""
May 26 11:47:28.975844 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:47:28.975843    2572 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ed85c58-ef71-4fde-b353-88d665ab6b39-dshm\") on node \"ip-10-0-136-194.ec2.internal\" DevicePath \"\""
May 26 11:47:28.975973 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:47:28.975853    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wvvsg\" (UniqueName: \"kubernetes.io/projected/1ed85c58-ef71-4fde-b353-88d665ab6b39-kube-api-access-wvvsg\") on node \"ip-10-0-136-194.ec2.internal\" DevicePath \"\""
May 26 11:47:28.975973 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:47:28.975863    2572 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ed85c58-ef71-4fde-b353-88d665ab6b39-home\") on node \"ip-10-0-136-194.ec2.internal\" DevicePath \"\""
May 26 11:47:28.975973 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:47:28.975871    2572 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ed85c58-ef71-4fde-b353-88d665ab6b39-tmp-dir\") on node \"ip-10-0-136-194.ec2.internal\" DevicePath \"\""
May 26 11:47:28.975973 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:47:28.975879    2572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ed85c58-ef71-4fde-b353-88d665ab6b39-tls-certs\") on node \"ip-10-0-136-194.ec2.internal\" DevicePath \"\""
May 26 11:47:29.409870 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:47:29.409843    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-b9b644d5d-54xv9_1ed85c58-ef71-4fde-b353-88d665ab6b39/main/0.log"
May 26 11:47:29.410174 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:47:29.410152    2572 generic.go:358] "Generic (PLEG): container finished" podID="1ed85c58-ef71-4fde-b353-88d665ab6b39" containerID="18c6e788cac9ba701a92001b7bf4cd9a11b0ff7bd1d9849a4277a49a93316783" exitCode=137
May 26 11:47:29.410268 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:47:29.410243    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-b9b644d5d-54xv9" event={"ID":"1ed85c58-ef71-4fde-b353-88d665ab6b39","Type":"ContainerDied","Data":"18c6e788cac9ba701a92001b7bf4cd9a11b0ff7bd1d9849a4277a49a93316783"}
May 26 11:47:29.410320 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:47:29.410288    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-b9b644d5d-54xv9" event={"ID":"1ed85c58-ef71-4fde-b353-88d665ab6b39","Type":"ContainerDied","Data":"7c4b4b966bff15425dd6319754cad895c11dd2f238daae73980c9c92b49bd691"}
May 26 11:47:29.410320 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:47:29.410305    2572 scope.go:117] "RemoveContainer" containerID="18c6e788cac9ba701a92001b7bf4cd9a11b0ff7bd1d9849a4277a49a93316783"
May 26 11:47:29.410393 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:47:29.410256    2572 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-b9b644d5d-54xv9"
May 26 11:47:29.426273 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:47:29.426252    2572 scope.go:117] "RemoveContainer" containerID="5882e40c86d22cff8bf3bb4ee77ba6726aa14ffa7cae7eccbddc378701aa79ad"
May 26 11:47:29.436745 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:47:29.436722    2572 scope.go:117] "RemoveContainer" containerID="18c6e788cac9ba701a92001b7bf4cd9a11b0ff7bd1d9849a4277a49a93316783"
May 26 11:47:29.437072 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:47:29.437041    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18c6e788cac9ba701a92001b7bf4cd9a11b0ff7bd1d9849a4277a49a93316783\": container with ID starting with 18c6e788cac9ba701a92001b7bf4cd9a11b0ff7bd1d9849a4277a49a93316783 not found: ID does not exist" containerID="18c6e788cac9ba701a92001b7bf4cd9a11b0ff7bd1d9849a4277a49a93316783"
May 26 11:47:29.437157 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:47:29.437077    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18c6e788cac9ba701a92001b7bf4cd9a11b0ff7bd1d9849a4277a49a93316783"} err="failed to get container status \"18c6e788cac9ba701a92001b7bf4cd9a11b0ff7bd1d9849a4277a49a93316783\": rpc error: code = NotFound desc = could not find container \"18c6e788cac9ba701a92001b7bf4cd9a11b0ff7bd1d9849a4277a49a93316783\": container with ID starting with 18c6e788cac9ba701a92001b7bf4cd9a11b0ff7bd1d9849a4277a49a93316783 not found: ID does not exist"
May 26 11:47:29.437157 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:47:29.437099    2572 scope.go:117] "RemoveContainer" containerID="5882e40c86d22cff8bf3bb4ee77ba6726aa14ffa7cae7eccbddc378701aa79ad"
May 26 11:47:29.437385 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:47:29.437367    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5882e40c86d22cff8bf3bb4ee77ba6726aa14ffa7cae7eccbddc378701aa79ad\": container with ID starting with 5882e40c86d22cff8bf3bb4ee77ba6726aa14ffa7cae7eccbddc378701aa79ad not found: ID does not exist" containerID="5882e40c86d22cff8bf3bb4ee77ba6726aa14ffa7cae7eccbddc378701aa79ad"
May 26 11:47:29.437433 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:47:29.437392    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5882e40c86d22cff8bf3bb4ee77ba6726aa14ffa7cae7eccbddc378701aa79ad"} err="failed to get container status \"5882e40c86d22cff8bf3bb4ee77ba6726aa14ffa7cae7eccbddc378701aa79ad\": rpc error: code = NotFound desc = could not find container \"5882e40c86d22cff8bf3bb4ee77ba6726aa14ffa7cae7eccbddc378701aa79ad\": container with ID starting with 5882e40c86d22cff8bf3bb4ee77ba6726aa14ffa7cae7eccbddc378701aa79ad not found: ID does not exist"
May 26 11:47:29.439159 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:47:29.439140    2572 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-b9b644d5d-54xv9"]
May 26 11:47:29.442721 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:47:29.442698    2572 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-55f7ae4a-kserve-b9b644d5d-54xv9"]
May 26 11:47:29.676776 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:47:29.676695    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1ed85c58-ef71-4fde-b353-88d665ab6b39" path="/var/lib/kubelet/pods/1ed85c58-ef71-4fde-b353-88d665ab6b39/volumes"
May 26 11:48:53.717500 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:48:53.717463    2572 generic.go:358] "Generic (PLEG): container finished" podID="2048d8b5-e1bc-41b0-bd27-8a8228cea5c5" containerID="b75ac10c88206a9c640d9656f13e05e5e8017429b34b55c805d21844a4e0ed36" exitCode=0
May 26 11:48:53.718088 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:48:53.717536    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-65d8bf8b78-5cdq2" event={"ID":"2048d8b5-e1bc-41b0-bd27-8a8228cea5c5","Type":"ContainerDied","Data":"b75ac10c88206a9c640d9656f13e05e5e8017429b34b55c805d21844a4e0ed36"}
May 26 11:48:54.722564 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:48:54.722521    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-65d8bf8b78-5cdq2" event={"ID":"2048d8b5-e1bc-41b0-bd27-8a8228cea5c5","Type":"ContainerStarted","Data":"fa7940035d3c0d3502d5f4087d1d35e1fbe852f2b81cc8673c7ffda0016fd07a"}
May 26 11:48:54.747833 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:48:54.747781    2572 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-65d8bf8b78-5cdq2" podStartSLOduration=102.74776505 podStartE2EDuration="1m42.74776505s" podCreationTimestamp="2026-05-26 11:47:12 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 11:48:54.746159856 +0000 UTC m=+1457.648370185" watchObservedRunningTime="2026-05-26 11:48:54.74776505 +0000 UTC m=+1457.649975379"
May 26 11:49:02.346767 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:49:02.346726    2572 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-65d8bf8b78-5cdq2"
May 26 11:49:02.346767 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:49:02.346773    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-65d8bf8b78-5cdq2"
May 26 11:49:02.348147 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:49:02.348117    2572 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-65d8bf8b78-5cdq2" podUID="2048d8b5-e1bc-41b0-bd27-8a8228cea5c5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/health\": dial tcp 10.132.0.43:8000: connect: connection refused"
May 26 11:49:12.346647 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:49:12.346603    2572 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-65d8bf8b78-5cdq2" podUID="2048d8b5-e1bc-41b0-bd27-8a8228cea5c5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/health\": dial tcp 10.132.0.43:8000: connect: connection refused"
May 26 11:49:22.347317 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:49:22.347258    2572 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-65d8bf8b78-5cdq2" podUID="2048d8b5-e1bc-41b0-bd27-8a8228cea5c5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/health\": dial tcp 10.132.0.43:8000: connect: connection refused"
May 26 11:49:32.346957 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:49:32.346904    2572 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-65d8bf8b78-5cdq2" podUID="2048d8b5-e1bc-41b0-bd27-8a8228cea5c5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/health\": dial tcp 10.132.0.43:8000: connect: connection refused"
May 26 11:49:37.681308 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:49:37.681279    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-5bfgg_c6373342-dd07-4497-a4f3-978ea9613b6e/console-operator/2.log"
May 26 11:49:37.686096 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:49:37.686058    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-5bfgg_c6373342-dd07-4497-a4f3-978ea9613b6e/console-operator/2.log"
May 26 11:49:37.688315 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:49:37.688292    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-js8cc_7264c907-f536-4719-905a-2bbda9e3c336/ovn-acl-logging/0.log"
May 26 11:49:37.692562 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:49:37.692546    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-js8cc_7264c907-f536-4719-905a-2bbda9e3c336/ovn-acl-logging/0.log"
May 26 11:49:42.347075 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:49:42.347030    2572 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-65d8bf8b78-5cdq2" podUID="2048d8b5-e1bc-41b0-bd27-8a8228cea5c5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/health\": dial tcp 10.132.0.43:8000: connect: connection refused"
May 26 11:49:52.347255 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:49:52.347215    2572 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-65d8bf8b78-5cdq2" podUID="2048d8b5-e1bc-41b0-bd27-8a8228cea5c5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/health\": dial tcp 10.132.0.43:8000: connect: connection refused"
May 26 11:50:02.347120 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:50:02.347065    2572 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-65d8bf8b78-5cdq2" podUID="2048d8b5-e1bc-41b0-bd27-8a8228cea5c5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/health\": dial tcp 10.132.0.43:8000: connect: connection refused"
May 26 11:50:12.347461 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:50:12.347410    2572 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-65d8bf8b78-5cdq2" podUID="2048d8b5-e1bc-41b0-bd27-8a8228cea5c5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/health\": dial tcp 10.132.0.43:8000: connect: connection refused"
May 26 11:50:22.347085 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:50:22.347028    2572 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-65d8bf8b78-5cdq2" podUID="2048d8b5-e1bc-41b0-bd27-8a8228cea5c5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/health\": dial tcp 10.132.0.43:8000: connect: connection refused"
May 26 11:50:32.356569 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:50:32.356534    2572 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-65d8bf8b78-5cdq2"
May 26 11:50:32.364321 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:50:32.364292    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-65d8bf8b78-5cdq2"
May 26 11:50:38.690207 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:50:38.690146    2572 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-65d8bf8b78-5cdq2"]
May 26 11:50:38.690686 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:50:38.690534    2572 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-65d8bf8b78-5cdq2" podUID="2048d8b5-e1bc-41b0-bd27-8a8228cea5c5" containerName="main" containerID="cri-o://fa7940035d3c0d3502d5f4087d1d35e1fbe852f2b81cc8673c7ffda0016fd07a" gracePeriod=30
May 26 11:50:47.342083 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:50:47.342050    2572 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-76745cd4f8-t96vl"]
May 26 11:50:47.342560 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:50:47.342463    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ed85c58-ef71-4fde-b353-88d665ab6b39" containerName="main"
May 26 11:50:47.342560 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:50:47.342477    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ed85c58-ef71-4fde-b353-88d665ab6b39" containerName="main"
May 26 11:50:47.342560 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:50:47.342510    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ed85c58-ef71-4fde-b353-88d665ab6b39" containerName="storage-initializer"
May 26 11:50:47.342560 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:50:47.342516    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ed85c58-ef71-4fde-b353-88d665ab6b39" containerName="storage-initializer"
May 26 11:50:47.342694 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:50:47.342578    2572 memory_manager.go:356] "RemoveStaleState removing state" podUID="1ed85c58-ef71-4fde-b353-88d665ab6b39" containerName="main"
May 26 11:50:47.346948 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:50:47.346930    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-76745cd4f8-t96vl"
May 26 11:50:47.349935 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:50:47.349915    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-with-refs-test-kserve-self-signed-certs\""
May 26 11:50:47.360565 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:50:47.360539    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-76745cd4f8-t96vl"]
May 26 11:50:47.425316 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:50:47.425285    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bcec67c7-b591-4630-b83d-1fffc2fd3dea-home\") pod \"router-with-refs-test-kserve-76745cd4f8-t96vl\" (UID: \"bcec67c7-b591-4630-b83d-1fffc2fd3dea\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-76745cd4f8-t96vl"
May 26 11:50:47.425459 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:50:47.425344    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bcec67c7-b591-4630-b83d-1fffc2fd3dea-dshm\") pod \"router-with-refs-test-kserve-76745cd4f8-t96vl\" (UID: \"bcec67c7-b591-4630-b83d-1fffc2fd3dea\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-76745cd4f8-t96vl"
May 26 11:50:47.425459 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:50:47.425368    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bcec67c7-b591-4630-b83d-1fffc2fd3dea-tmp-dir\") pod \"router-with-refs-test-kserve-76745cd4f8-t96vl\" (UID: \"bcec67c7-b591-4630-b83d-1fffc2fd3dea\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-76745cd4f8-t96vl"
May 26 11:50:47.425459 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:50:47.425415    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bcec67c7-b591-4630-b83d-1fffc2fd3dea-model-cache\") pod \"router-with-refs-test-kserve-76745cd4f8-t96vl\" (UID: \"bcec67c7-b591-4630-b83d-1fffc2fd3dea\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-76745cd4f8-t96vl"
May 26 11:50:47.425564 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:50:47.425457    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6x9z\" (UniqueName: \"kubernetes.io/projected/bcec67c7-b591-4630-b83d-1fffc2fd3dea-kube-api-access-f6x9z\") pod \"router-with-refs-test-kserve-76745cd4f8-t96vl\" (UID: \"bcec67c7-b591-4630-b83d-1fffc2fd3dea\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-76745cd4f8-t96vl"
May 26 11:50:47.425564 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:50:47.425519    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bcec67c7-b591-4630-b83d-1fffc2fd3dea-kserve-provision-location\") pod \"router-with-refs-test-kserve-76745cd4f8-t96vl\" (UID: \"bcec67c7-b591-4630-b83d-1fffc2fd3dea\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-76745cd4f8-t96vl"
May 26 11:50:47.425564 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:50:47.425543    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bcec67c7-b591-4630-b83d-1fffc2fd3dea-tls-certs\") pod \"router-with-refs-test-kserve-76745cd4f8-t96vl\" (UID: \"bcec67c7-b591-4630-b83d-1fffc2fd3dea\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-76745cd4f8-t96vl"
May 26 11:50:47.527029 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:50:47.526996    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bcec67c7-b591-4630-b83d-1fffc2fd3dea-dshm\") pod \"router-with-refs-test-kserve-76745cd4f8-t96vl\" (UID: \"bcec67c7-b591-4630-b83d-1fffc2fd3dea\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-76745cd4f8-t96vl"
May 26 11:50:47.527268 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:50:47.527041    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bcec67c7-b591-4630-b83d-1fffc2fd3dea-tmp-dir\") pod \"router-with-refs-test-kserve-76745cd4f8-t96vl\" (UID: \"bcec67c7-b591-4630-b83d-1fffc2fd3dea\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-76745cd4f8-t96vl"
May 26 11:50:47.527268 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:50:47.527102    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bcec67c7-b591-4630-b83d-1fffc2fd3dea-model-cache\") pod \"router-with-refs-test-kserve-76745cd4f8-t96vl\" (UID: \"bcec67c7-b591-4630-b83d-1fffc2fd3dea\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-76745cd4f8-t96vl"
May 26 11:50:47.527268 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:50:47.527133    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-f6x9z\" (UniqueName: \"kubernetes.io/projected/bcec67c7-b591-4630-b83d-1fffc2fd3dea-kube-api-access-f6x9z\") pod \"router-with-refs-test-kserve-76745cd4f8-t96vl\" (UID: \"bcec67c7-b591-4630-b83d-1fffc2fd3dea\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-76745cd4f8-t96vl"
May 26 11:50:47.527268 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:50:47.527233    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bcec67c7-b591-4630-b83d-1fffc2fd3dea-kserve-provision-location\") pod \"router-with-refs-test-kserve-76745cd4f8-t96vl\" (UID: \"bcec67c7-b591-4630-b83d-1fffc2fd3dea\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-76745cd4f8-t96vl"
May 26 11:50:47.527268 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:50:47.527267    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bcec67c7-b591-4630-b83d-1fffc2fd3dea-tls-certs\") pod \"router-with-refs-test-kserve-76745cd4f8-t96vl\" (UID: \"bcec67c7-b591-4630-b83d-1fffc2fd3dea\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-76745cd4f8-t96vl"
May 26 11:50:47.527541 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:50:47.527310    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bcec67c7-b591-4630-b83d-1fffc2fd3dea-home\") pod \"router-with-refs-test-kserve-76745cd4f8-t96vl\" (UID: \"bcec67c7-b591-4630-b83d-1fffc2fd3dea\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-76745cd4f8-t96vl"
May 26 11:50:47.527541 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:50:47.527487    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bcec67c7-b591-4630-b83d-1fffc2fd3dea-tmp-dir\") pod \"router-with-refs-test-kserve-76745cd4f8-t96vl\" (UID: \"bcec67c7-b591-4630-b83d-1fffc2fd3dea\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-76745cd4f8-t96vl"
May 26 11:50:47.527656 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:50:47.527596    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bcec67c7-b591-4630-b83d-1fffc2fd3dea-model-cache\") pod \"router-with-refs-test-kserve-76745cd4f8-t96vl\" (UID: \"bcec67c7-b591-4630-b83d-1fffc2fd3dea\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-76745cd4f8-t96vl"
May 26 11:50:47.527716 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:50:47.527671    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bcec67c7-b591-4630-b83d-1fffc2fd3dea-kserve-provision-location\") pod \"router-with-refs-test-kserve-76745cd4f8-t96vl\" (UID: \"bcec67c7-b591-4630-b83d-1fffc2fd3dea\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-76745cd4f8-t96vl"
May 26 11:50:47.527773 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:50:47.527712    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bcec67c7-b591-4630-b83d-1fffc2fd3dea-home\") pod \"router-with-refs-test-kserve-76745cd4f8-t96vl\" (UID: \"bcec67c7-b591-4630-b83d-1fffc2fd3dea\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-76745cd4f8-t96vl"
May 26 11:50:47.529379 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:50:47.529360    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bcec67c7-b591-4630-b83d-1fffc2fd3dea-dshm\") pod \"router-with-refs-test-kserve-76745cd4f8-t96vl\" (UID: \"bcec67c7-b591-4630-b83d-1fffc2fd3dea\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-76745cd4f8-t96vl"
May 26 11:50:47.529862 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:50:47.529830    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bcec67c7-b591-4630-b83d-1fffc2fd3dea-tls-certs\") pod \"router-with-refs-test-kserve-76745cd4f8-t96vl\" (UID: \"bcec67c7-b591-4630-b83d-1fffc2fd3dea\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-76745cd4f8-t96vl"
May 26 11:50:47.535711 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:50:47.535683    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6x9z\" (UniqueName: \"kubernetes.io/projected/bcec67c7-b591-4630-b83d-1fffc2fd3dea-kube-api-access-f6x9z\") pod \"router-with-refs-test-kserve-76745cd4f8-t96vl\" (UID: \"bcec67c7-b591-4630-b83d-1fffc2fd3dea\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-76745cd4f8-t96vl"
May 26 11:50:47.657281 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:50:47.657246    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-76745cd4f8-t96vl"
May 26 11:50:47.803705 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:50:47.803669    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-76745cd4f8-t96vl"]
May 26 11:50:47.806670 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:50:47.806641    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcec67c7_b591_4630_b83d_1fffc2fd3dea.slice/crio-04a40586fd4fbf89d53733d26335f0a87d61a01a906117d81cf5b20223bfc16d WatchSource:0}: Error finding container 04a40586fd4fbf89d53733d26335f0a87d61a01a906117d81cf5b20223bfc16d: Status 404 returned error can't find the container with id 04a40586fd4fbf89d53733d26335f0a87d61a01a906117d81cf5b20223bfc16d
May 26 11:50:47.808982 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:50:47.808964    2572 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 26 11:50:48.147073 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:50:48.146983    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-76745cd4f8-t96vl" event={"ID":"bcec67c7-b591-4630-b83d-1fffc2fd3dea","Type":"ContainerStarted","Data":"0197db812af5d2361920da7a0bbbf37d0767f588740349342270d1feae5733c3"}
May 26 11:50:48.147073 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:50:48.147019    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-76745cd4f8-t96vl" event={"ID":"bcec67c7-b591-4630-b83d-1fffc2fd3dea","Type":"ContainerStarted","Data":"04a40586fd4fbf89d53733d26335f0a87d61a01a906117d81cf5b20223bfc16d"}
May 26 11:50:53.169126 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:50:53.169091    2572 generic.go:358] "Generic (PLEG): container finished" podID="bcec67c7-b591-4630-b83d-1fffc2fd3dea" containerID="0197db812af5d2361920da7a0bbbf37d0767f588740349342270d1feae5733c3" exitCode=0
May 26 11:50:53.169524 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:50:53.169138    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-76745cd4f8-t96vl" event={"ID":"bcec67c7-b591-4630-b83d-1fffc2fd3dea","Type":"ContainerDied","Data":"0197db812af5d2361920da7a0bbbf37d0767f588740349342270d1feae5733c3"}
May 26 11:50:54.175052 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:50:54.175019    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-76745cd4f8-t96vl" event={"ID":"bcec67c7-b591-4630-b83d-1fffc2fd3dea","Type":"ContainerStarted","Data":"0e65dac81bc1976de0cb18a2c44fe5d203bcc4cbe41fb52f929ce79b2443b153"}
May 26 11:50:54.196844 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:50:54.196793    2572 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-76745cd4f8-t96vl" podStartSLOduration=7.196777796 podStartE2EDuration="7.196777796s" podCreationTimestamp="2026-05-26 11:50:47 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 11:50:54.195263154 +0000 UTC m=+1577.097473486" watchObservedRunningTime="2026-05-26 11:50:54.196777796 +0000 UTC m=+1577.098988125"
May 26 11:50:57.657867 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:50:57.657832    2572 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-76745cd4f8-t96vl"
May 26 11:50:57.657867 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:50:57.657876    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-76745cd4f8-t96vl"
May 26 11:50:57.659453 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:50:57.659424    2572 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-76745cd4f8-t96vl" podUID="bcec67c7-b591-4630-b83d-1fffc2fd3dea" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/health\": dial tcp 10.132.0.44:8000: connect: connection refused"
May 26 11:51:07.658710 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:51:07.658666    2572 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-76745cd4f8-t96vl" podUID="bcec67c7-b591-4630-b83d-1fffc2fd3dea" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/health\": dial tcp 10.132.0.44:8000: connect: connection refused"
May 26 11:51:08.958165 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:51:08.958101    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_custom-route-timeout-test-kserve-65d8bf8b78-5cdq2_2048d8b5-e1bc-41b0-bd27-8a8228cea5c5/main/0.log"
May 26 11:51:08.958533 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:51:08.958518    2572 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-65d8bf8b78-5cdq2"
May 26 11:51:09.031459 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:51:09.031426    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2048d8b5-e1bc-41b0-bd27-8a8228cea5c5-kserve-provision-location\") pod \"2048d8b5-e1bc-41b0-bd27-8a8228cea5c5\" (UID: \"2048d8b5-e1bc-41b0-bd27-8a8228cea5c5\") "
May 26 11:51:09.031639 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:51:09.031467    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2048d8b5-e1bc-41b0-bd27-8a8228cea5c5-tmp-dir\") pod \"2048d8b5-e1bc-41b0-bd27-8a8228cea5c5\" (UID: \"2048d8b5-e1bc-41b0-bd27-8a8228cea5c5\") "
May 26 11:51:09.031639 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:51:09.031498    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2048d8b5-e1bc-41b0-bd27-8a8228cea5c5-model-cache\") pod \"2048d8b5-e1bc-41b0-bd27-8a8228cea5c5\" (UID: \"2048d8b5-e1bc-41b0-bd27-8a8228cea5c5\") "
May 26 11:51:09.031732 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:51:09.031684    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2048d8b5-e1bc-41b0-bd27-8a8228cea5c5-tls-certs\") pod \"2048d8b5-e1bc-41b0-bd27-8a8228cea5c5\" (UID: \"2048d8b5-e1bc-41b0-bd27-8a8228cea5c5\") "
May 26 11:51:09.031732 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:51:09.031707    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2048d8b5-e1bc-41b0-bd27-8a8228cea5c5-model-cache" (OuterVolumeSpecName: "model-cache") pod "2048d8b5-e1bc-41b0-bd27-8a8228cea5c5" (UID: "2048d8b5-e1bc-41b0-bd27-8a8228cea5c5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 11:51:09.031732 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:51:09.031728    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2048d8b5-e1bc-41b0-bd27-8a8228cea5c5-dshm\") pod \"2048d8b5-e1bc-41b0-bd27-8a8228cea5c5\" (UID: \"2048d8b5-e1bc-41b0-bd27-8a8228cea5c5\") "
May 26 11:51:09.031898 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:51:09.031785    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2048d8b5-e1bc-41b0-bd27-8a8228cea5c5-home\") pod \"2048d8b5-e1bc-41b0-bd27-8a8228cea5c5\" (UID: \"2048d8b5-e1bc-41b0-bd27-8a8228cea5c5\") "
May 26 11:51:09.031898 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:51:09.031817    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nb4jn\" (UniqueName: \"kubernetes.io/projected/2048d8b5-e1bc-41b0-bd27-8a8228cea5c5-kube-api-access-nb4jn\") pod \"2048d8b5-e1bc-41b0-bd27-8a8228cea5c5\" (UID: \"2048d8b5-e1bc-41b0-bd27-8a8228cea5c5\") "
May 26 11:51:09.032222 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:51:09.032180    2572 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2048d8b5-e1bc-41b0-bd27-8a8228cea5c5-model-cache\") on node \"ip-10-0-136-194.ec2.internal\" DevicePath \"\""
May 26 11:51:09.032475 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:51:09.032405    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2048d8b5-e1bc-41b0-bd27-8a8228cea5c5-home" (OuterVolumeSpecName: "home") pod "2048d8b5-e1bc-41b0-bd27-8a8228cea5c5" (UID: "2048d8b5-e1bc-41b0-bd27-8a8228cea5c5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 11:51:09.034155 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:51:09.034127    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2048d8b5-e1bc-41b0-bd27-8a8228cea5c5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2048d8b5-e1bc-41b0-bd27-8a8228cea5c5" (UID: "2048d8b5-e1bc-41b0-bd27-8a8228cea5c5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 11:51:09.034382 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:51:09.034363    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2048d8b5-e1bc-41b0-bd27-8a8228cea5c5-dshm" (OuterVolumeSpecName: "dshm") pod "2048d8b5-e1bc-41b0-bd27-8a8228cea5c5" (UID: "2048d8b5-e1bc-41b0-bd27-8a8228cea5c5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 11:51:09.034593 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:51:09.034573    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2048d8b5-e1bc-41b0-bd27-8a8228cea5c5-kube-api-access-nb4jn" (OuterVolumeSpecName: "kube-api-access-nb4jn") pod "2048d8b5-e1bc-41b0-bd27-8a8228cea5c5" (UID: "2048d8b5-e1bc-41b0-bd27-8a8228cea5c5"). InnerVolumeSpecName "kube-api-access-nb4jn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 11:51:09.044047 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:51:09.044026    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2048d8b5-e1bc-41b0-bd27-8a8228cea5c5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2048d8b5-e1bc-41b0-bd27-8a8228cea5c5" (UID: "2048d8b5-e1bc-41b0-bd27-8a8228cea5c5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 11:51:09.094151 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:51:09.094111    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2048d8b5-e1bc-41b0-bd27-8a8228cea5c5-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2048d8b5-e1bc-41b0-bd27-8a8228cea5c5" (UID: "2048d8b5-e1bc-41b0-bd27-8a8228cea5c5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 11:51:09.133240 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:51:09.133215    2572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2048d8b5-e1bc-41b0-bd27-8a8228cea5c5-tls-certs\") on node \"ip-10-0-136-194.ec2.internal\" DevicePath \"\""
May 26 11:51:09.133240 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:51:09.133240    2572 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2048d8b5-e1bc-41b0-bd27-8a8228cea5c5-dshm\") on node \"ip-10-0-136-194.ec2.internal\" DevicePath \"\""
May 26 11:51:09.133404 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:51:09.133249    2572 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2048d8b5-e1bc-41b0-bd27-8a8228cea5c5-home\") on node \"ip-10-0-136-194.ec2.internal\" DevicePath \"\""
May 26 11:51:09.133404 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:51:09.133258    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nb4jn\" (UniqueName: \"kubernetes.io/projected/2048d8b5-e1bc-41b0-bd27-8a8228cea5c5-kube-api-access-nb4jn\") on node \"ip-10-0-136-194.ec2.internal\" DevicePath \"\""
May 26 11:51:09.133404 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:51:09.133269    2572 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2048d8b5-e1bc-41b0-bd27-8a8228cea5c5-kserve-provision-location\") on node \"ip-10-0-136-194.ec2.internal\" DevicePath \"\""
May 26 11:51:09.133404 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:51:09.133278    2572 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2048d8b5-e1bc-41b0-bd27-8a8228cea5c5-tmp-dir\") on node \"ip-10-0-136-194.ec2.internal\" DevicePath \"\""
May 26 11:51:09.233369 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:51:09.233284    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_custom-route-timeout-test-kserve-65d8bf8b78-5cdq2_2048d8b5-e1bc-41b0-bd27-8a8228cea5c5/main/0.log"
May 26 11:51:09.233676 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:51:09.233651    2572 generic.go:358] "Generic (PLEG): container finished" podID="2048d8b5-e1bc-41b0-bd27-8a8228cea5c5" containerID="fa7940035d3c0d3502d5f4087d1d35e1fbe852f2b81cc8673c7ffda0016fd07a" exitCode=137
May 26 11:51:09.233785 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:51:09.233721    2572 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-65d8bf8b78-5cdq2"
May 26 11:51:09.233785 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:51:09.233747    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-65d8bf8b78-5cdq2" event={"ID":"2048d8b5-e1bc-41b0-bd27-8a8228cea5c5","Type":"ContainerDied","Data":"fa7940035d3c0d3502d5f4087d1d35e1fbe852f2b81cc8673c7ffda0016fd07a"}
May 26 11:51:09.233880 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:51:09.233815    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-65d8bf8b78-5cdq2" event={"ID":"2048d8b5-e1bc-41b0-bd27-8a8228cea5c5","Type":"ContainerDied","Data":"57825cdfc31aee39de0a7b6c28cc8fcb8bef124ca5b897f7efc4da9a9ae44069"}
May 26 11:51:09.233880 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:51:09.233841    2572 scope.go:117] "RemoveContainer" containerID="fa7940035d3c0d3502d5f4087d1d35e1fbe852f2b81cc8673c7ffda0016fd07a"
May 26 11:51:09.243133 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:51:09.243114    2572 scope.go:117] "RemoveContainer" containerID="b75ac10c88206a9c640d9656f13e05e5e8017429b34b55c805d21844a4e0ed36"
May 26 11:51:09.258509 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:51:09.258481    2572 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-65d8bf8b78-5cdq2"]
May 26 11:51:09.260060 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:51:09.260033    2572 scope.go:117] "RemoveContainer" containerID="fa7940035d3c0d3502d5f4087d1d35e1fbe852f2b81cc8673c7ffda0016fd07a"
May 26 11:51:09.260437 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:51:09.260411    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa7940035d3c0d3502d5f4087d1d35e1fbe852f2b81cc8673c7ffda0016fd07a\": container with ID starting with fa7940035d3c0d3502d5f4087d1d35e1fbe852f2b81cc8673c7ffda0016fd07a not found: ID does not exist" containerID="fa7940035d3c0d3502d5f4087d1d35e1fbe852f2b81cc8673c7ffda0016fd07a"
May 26 11:51:09.260535 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:51:09.260447    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa7940035d3c0d3502d5f4087d1d35e1fbe852f2b81cc8673c7ffda0016fd07a"} err="failed to get container status \"fa7940035d3c0d3502d5f4087d1d35e1fbe852f2b81cc8673c7ffda0016fd07a\": rpc error: code = NotFound desc = could not find container \"fa7940035d3c0d3502d5f4087d1d35e1fbe852f2b81cc8673c7ffda0016fd07a\": container with ID starting with fa7940035d3c0d3502d5f4087d1d35e1fbe852f2b81cc8673c7ffda0016fd07a not found: ID does not exist"
May 26 11:51:09.260535 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:51:09.260477    2572 scope.go:117] "RemoveContainer" containerID="b75ac10c88206a9c640d9656f13e05e5e8017429b34b55c805d21844a4e0ed36"
May 26 11:51:09.260726 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:51:09.260703    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b75ac10c88206a9c640d9656f13e05e5e8017429b34b55c805d21844a4e0ed36\": container with ID starting with b75ac10c88206a9c640d9656f13e05e5e8017429b34b55c805d21844a4e0ed36 not found: ID does not exist" containerID="b75ac10c88206a9c640d9656f13e05e5e8017429b34b55c805d21844a4e0ed36"
May 26 11:51:09.260788 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:51:09.260744    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b75ac10c88206a9c640d9656f13e05e5e8017429b34b55c805d21844a4e0ed36"} err="failed to get container status \"b75ac10c88206a9c640d9656f13e05e5e8017429b34b55c805d21844a4e0ed36\": rpc error: code = NotFound desc = could not find container \"b75ac10c88206a9c640d9656f13e05e5e8017429b34b55c805d21844a4e0ed36\": container with ID starting with b75ac10c88206a9c640d9656f13e05e5e8017429b34b55c805d21844a4e0ed36 not found: ID does not exist"
May 26 11:51:09.261809 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:51:09.261791    2572 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-65d8bf8b78-5cdq2"]
May 26 11:51:09.676578 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:51:09.676547    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2048d8b5-e1bc-41b0-bd27-8a8228cea5c5" path="/var/lib/kubelet/pods/2048d8b5-e1bc-41b0-bd27-8a8228cea5c5/volumes"
May 26 11:51:17.658012 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:51:17.657972    2572 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-76745cd4f8-t96vl" podUID="bcec67c7-b591-4630-b83d-1fffc2fd3dea" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/health\": dial tcp 10.132.0.44:8000: connect: connection refused"
May 26 11:51:27.658087 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:51:27.657981    2572 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-76745cd4f8-t96vl" podUID="bcec67c7-b591-4630-b83d-1fffc2fd3dea" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/health\": dial tcp 10.132.0.44:8000: connect: connection refused"
May 26 11:51:37.658669 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:51:37.658629    2572 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-76745cd4f8-t96vl" podUID="bcec67c7-b591-4630-b83d-1fffc2fd3dea" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/health\": dial tcp 10.132.0.44:8000: connect: connection refused"
May 26 11:51:47.658650 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:51:47.658605    2572 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-76745cd4f8-t96vl" podUID="bcec67c7-b591-4630-b83d-1fffc2fd3dea" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/health\": dial tcp 10.132.0.44:8000: connect: connection refused"
May 26 11:51:57.658070 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:51:57.658024    2572 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-76745cd4f8-t96vl" podUID="bcec67c7-b591-4630-b83d-1fffc2fd3dea" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/health\": dial tcp 10.132.0.44:8000: connect: connection refused"
May 26 11:52:07.657963 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:52:07.657913    2572 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-76745cd4f8-t96vl" podUID="bcec67c7-b591-4630-b83d-1fffc2fd3dea" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/health\": dial tcp 10.132.0.44:8000: connect: connection refused"
May 26 11:52:17.658069 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:52:17.658023    2572 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-76745cd4f8-t96vl" podUID="bcec67c7-b591-4630-b83d-1fffc2fd3dea" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/health\": dial tcp 10.132.0.44:8000: connect: connection refused"
May 26 11:52:27.667729 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:52:27.667701    2572 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-76745cd4f8-t96vl"
May 26 11:52:27.675860 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:52:27.675835    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-76745cd4f8-t96vl"
May 26 11:52:34.129586 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:52:34.129545    2572 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-76745cd4f8-t96vl"]
May 26 11:52:34.130081 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:52:34.129848    2572 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-76745cd4f8-t96vl" podUID="bcec67c7-b591-4630-b83d-1fffc2fd3dea" containerName="main" containerID="cri-o://0e65dac81bc1976de0cb18a2c44fe5d203bcc4cbe41fb52f929ce79b2443b153" gracePeriod=30
May 26 11:52:44.362361 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:52:44.362325    2572 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7bc7c9f5cf8xqlh"]
May 26 11:52:44.362724 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:52:44.362698    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2048d8b5-e1bc-41b0-bd27-8a8228cea5c5" containerName="main"
May 26 11:52:44.362724 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:52:44.362709    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="2048d8b5-e1bc-41b0-bd27-8a8228cea5c5" containerName="main"
May 26 11:52:44.362724 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:52:44.362717    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2048d8b5-e1bc-41b0-bd27-8a8228cea5c5" containerName="storage-initializer"
May 26 11:52:44.362724 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:52:44.362723    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="2048d8b5-e1bc-41b0-bd27-8a8228cea5c5" containerName="storage-initializer"
May 26 11:52:44.362859 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:52:44.362803    2572 memory_manager.go:356] "RemoveStaleState removing state" podUID="2048d8b5-e1bc-41b0-bd27-8a8228cea5c5" containerName="main"
May 26 11:52:44.366147 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:52:44.366131    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7bc7c9f5cf8xqlh"
May 26 11:52:44.369127 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:52:44.369103    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-dockercfg-tgn7g\""
May 26 11:52:44.369275 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:52:44.369131    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv8f1a6f044e8c7a4d31a250e0c4861caf-kserve-self-signed-certs\""
May 26 11:52:44.376682 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:52:44.376658    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7bc7c9f5cf8xqlh"]
May 26 11:52:44.381752 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:52:44.381734    2572 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-55hg57h"]
May 26 11:52:44.385314 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:52:44.385297    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-55hg57h"
May 26 11:52:44.396232 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:52:44.396178    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-55hg57h"]
May 26 11:52:44.558175 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:52:44.558136    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/25b0394c-f00b-4f2e-acb4-853d2a617f79-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7bc7c9f5cf8xqlh\" (UID: \"25b0394c-f00b-4f2e-acb4-853d2a617f79\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7bc7c9f5cf8xqlh"
May 26 11:52:44.558398 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:52:44.558176    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/25b0394c-f00b-4f2e-acb4-853d2a617f79-home\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7bc7c9f5cf8xqlh\" (UID: \"25b0394c-f00b-4f2e-acb4-853d2a617f79\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7bc7c9f5cf8xqlh"
May 26 11:52:44.558398 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:52:44.558254    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/25b0394c-f00b-4f2e-acb4-853d2a617f79-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7bc7c9f5cf8xqlh\" (UID: \"25b0394c-f00b-4f2e-acb4-853d2a617f79\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7bc7c9f5cf8xqlh"
May 26 11:52:44.558398 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:52:44.558297    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/25b0394c-f00b-4f2e-acb4-853d2a617f79-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7bc7c9f5cf8xqlh\" (UID: \"25b0394c-f00b-4f2e-acb4-853d2a617f79\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7bc7c9f5cf8xqlh"
May 26 11:52:44.558398 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:52:44.558367    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61e6b55c-5d96-4566-8f1a-c35a4097c57f-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-55hg57h\" (UID: \"61e6b55c-5d96-4566-8f1a-c35a4097c57f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-55hg57h"
May 26 11:52:44.558398 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:52:44.558394    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/61e6b55c-5d96-4566-8f1a-c35a4097c57f-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-55hg57h\" (UID: \"61e6b55c-5d96-4566-8f1a-c35a4097c57f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-55hg57h"
May 26 11:52:44.558678 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:52:44.558422    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/25b0394c-f00b-4f2e-acb4-853d2a617f79-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7bc7c9f5cf8xqlh\" (UID: \"25b0394c-f00b-4f2e-acb4-853d2a617f79\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7bc7c9f5cf8xqlh"
May 26 11:52:44.558678 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:52:44.558452    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/61e6b55c-5d96-4566-8f1a-c35a4097c57f-dshm\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-55hg57h\" (UID: \"61e6b55c-5d96-4566-8f1a-c35a4097c57f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-55hg57h"
May 26 11:52:44.558678 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:52:44.558481    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/61e6b55c-5d96-4566-8f1a-c35a4097c57f-home\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-55hg57h\" (UID: \"61e6b55c-5d96-4566-8f1a-c35a4097c57f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-55hg57h"
May 26 11:52:44.558678 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:52:44.558529    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9qhj\" (UniqueName: \"kubernetes.io/projected/25b0394c-f00b-4f2e-acb4-853d2a617f79-kube-api-access-d9qhj\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7bc7c9f5cf8xqlh\" (UID: \"25b0394c-f00b-4f2e-acb4-853d2a617f79\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7bc7c9f5cf8xqlh"
May 26 11:52:44.558678 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:52:44.558569    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lf4rh\" (UniqueName: \"kubernetes.io/projected/61e6b55c-5d96-4566-8f1a-c35a4097c57f-kube-api-access-lf4rh\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-55hg57h\" (UID: \"61e6b55c-5d96-4566-8f1a-c35a4097c57f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-55hg57h"
May 26 11:52:44.558678 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:52:44.558612    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61e6b55c-5d96-4566-8f1a-c35a4097c57f-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-55hg57h\" (UID: \"61e6b55c-5d96-4566-8f1a-c35a4097c57f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-55hg57h"
May 26 11:52:44.558678 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:52:44.558643    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61e6b55c-5d96-4566-8f1a-c35a4097c57f-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-55hg57h\" (UID: \"61e6b55c-5d96-4566-8f1a-c35a4097c57f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-55hg57h"
May 26 11:52:44.558999 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:52:44.558711    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/25b0394c-f00b-4f2e-acb4-853d2a617f79-dshm\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7bc7c9f5cf8xqlh\" (UID: \"25b0394c-f00b-4f2e-acb4-853d2a617f79\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7bc7c9f5cf8xqlh"
May 26 11:52:44.660117 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:52:44.660086    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/25b0394c-f00b-4f2e-acb4-853d2a617f79-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7bc7c9f5cf8xqlh\" (UID: \"25b0394c-f00b-4f2e-acb4-853d2a617f79\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7bc7c9f5cf8xqlh"
May 26 11:52:44.660337 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:52:44.660125    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/25b0394c-f00b-4f2e-acb4-853d2a617f79-home\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7bc7c9f5cf8xqlh\" (UID: \"25b0394c-f00b-4f2e-acb4-853d2a617f79\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7bc7c9f5cf8xqlh"
May 26 11:52:44.660337 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:52:44.660151    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/25b0394c-f00b-4f2e-acb4-853d2a617f79-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7bc7c9f5cf8xqlh\" (UID: \"25b0394c-f00b-4f2e-acb4-853d2a617f79\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7bc7c9f5cf8xqlh"
May 26 11:52:44.660337 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:52:44.660177    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/25b0394c-f00b-4f2e-acb4-853d2a617f79-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7bc7c9f5cf8xqlh\" (UID: \"25b0394c-f00b-4f2e-acb4-853d2a617f79\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7bc7c9f5cf8xqlh"
May 26 11:52:44.660337 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:52:44.660277    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61e6b55c-5d96-4566-8f1a-c35a4097c57f-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-55hg57h\" (UID: \"61e6b55c-5d96-4566-8f1a-c35a4097c57f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-55hg57h"
May 26 11:52:44.660337 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:52:44.660302    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/61e6b55c-5d96-4566-8f1a-c35a4097c57f-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-55hg57h\" (UID: \"61e6b55c-5d96-4566-8f1a-c35a4097c57f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-55hg57h"
May 26 11:52:44.660337 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:52:44.660326    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/25b0394c-f00b-4f2e-acb4-853d2a617f79-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7bc7c9f5cf8xqlh\" (UID: \"25b0394c-f00b-4f2e-acb4-853d2a617f79\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7bc7c9f5cf8xqlh"
May 26 11:52:44.660651 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:52:44.660354    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/61e6b55c-5d96-4566-8f1a-c35a4097c57f-dshm\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-55hg57h\" (UID: \"61e6b55c-5d96-4566-8f1a-c35a4097c57f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-55hg57h"
May 26 11:52:44.660651 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:52:44.660381    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/61e6b55c-5d96-4566-8f1a-c35a4097c57f-home\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-55hg57h\" (UID: \"61e6b55c-5d96-4566-8f1a-c35a4097c57f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-55hg57h"
May 26 11:52:44.660651 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:52:44.660420    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-d9qhj\" (UniqueName: \"kubernetes.io/projected/25b0394c-f00b-4f2e-acb4-853d2a617f79-kube-api-access-d9qhj\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7bc7c9f5cf8xqlh\" (UID: \"25b0394c-f00b-4f2e-acb4-853d2a617f79\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7bc7c9f5cf8xqlh"
May 26 11:52:44.660651 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:52:44.660458    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-lf4rh\" (UniqueName: \"kubernetes.io/projected/61e6b55c-5d96-4566-8f1a-c35a4097c57f-kube-api-access-lf4rh\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-55hg57h\" (UID: \"61e6b55c-5d96-4566-8f1a-c35a4097c57f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-55hg57h"
May 26 11:52:44.660651 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:52:44.660504    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61e6b55c-5d96-4566-8f1a-c35a4097c57f-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-55hg57h\" (UID: \"61e6b55c-5d96-4566-8f1a-c35a4097c57f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-55hg57h"
May 26 11:52:44.660651 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:52:44.660537    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61e6b55c-5d96-4566-8f1a-c35a4097c57f-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-55hg57h\" (UID: \"61e6b55c-5d96-4566-8f1a-c35a4097c57f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-55hg57h"
May 26 11:52:44.660651 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:52:44.660579    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/25b0394c-f00b-4f2e-acb4-853d2a617f79-dshm\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7bc7c9f5cf8xqlh\" (UID: \"25b0394c-f00b-4f2e-acb4-853d2a617f79\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7bc7c9f5cf8xqlh"
May 26 11:52:44.660651 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:52:44.660596    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/25b0394c-f00b-4f2e-acb4-853d2a617f79-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7bc7c9f5cf8xqlh\" (UID: \"25b0394c-f00b-4f2e-acb4-853d2a617f79\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7bc7c9f5cf8xqlh"
May 26 11:52:44.660651 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:52:44.660623    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/25b0394c-f00b-4f2e-acb4-853d2a617f79-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7bc7c9f5cf8xqlh\" (UID: \"25b0394c-f00b-4f2e-acb4-853d2a617f79\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7bc7c9f5cf8xqlh"
May 26 11:52:44.661049 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:52:44.660674    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/25b0394c-f00b-4f2e-acb4-853d2a617f79-home\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7bc7c9f5cf8xqlh\" (UID: \"25b0394c-f00b-4f2e-acb4-853d2a617f79\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7bc7c9f5cf8xqlh"
May 26 11:52:44.661049 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:52:44.660717    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61e6b55c-5d96-4566-8f1a-c35a4097c57f-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-55hg57h\" (UID: \"61e6b55c-5d96-4566-8f1a-c35a4097c57f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-55hg57h"
May 26 11:52:44.661049 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:52:44.660760    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/61e6b55c-5d96-4566-8f1a-c35a4097c57f-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-55hg57h\" (UID: \"61e6b55c-5d96-4566-8f1a-c35a4097c57f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-55hg57h"
May 26 11:52:44.661049 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:52:44.661001    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/25b0394c-f00b-4f2e-acb4-853d2a617f79-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7bc7c9f5cf8xqlh\" (UID: \"25b0394c-f00b-4f2e-acb4-853d2a617f79\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7bc7c9f5cf8xqlh"
May 26 11:52:44.661049 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:52:44.661042    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61e6b55c-5d96-4566-8f1a-c35a4097c57f-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-55hg57h\" (UID: \"61e6b55c-5d96-4566-8f1a-c35a4097c57f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-55hg57h"
May 26 11:52:44.661344 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:52:44.661105    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/61e6b55c-5d96-4566-8f1a-c35a4097c57f-home\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-55hg57h\" (UID: \"61e6b55c-5d96-4566-8f1a-c35a4097c57f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-55hg57h"
May 26 11:52:44.663000 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:52:44.662975    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/25b0394c-f00b-4f2e-acb4-853d2a617f79-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7bc7c9f5cf8xqlh\" (UID: \"25b0394c-f00b-4f2e-acb4-853d2a617f79\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7bc7c9f5cf8xqlh"
May 26 11:52:44.663134 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:52:44.662999    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/25b0394c-f00b-4f2e-acb4-853d2a617f79-dshm\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7bc7c9f5cf8xqlh\" (UID: \"25b0394c-f00b-4f2e-acb4-853d2a617f79\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7bc7c9f5cf8xqlh"
May 26 11:52:44.663134 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:52:44.663062    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/61e6b55c-5d96-4566-8f1a-c35a4097c57f-dshm\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-55hg57h\" (UID: \"61e6b55c-5d96-4566-8f1a-c35a4097c57f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-55hg57h"
May 26 11:52:44.663283 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:52:44.663265    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61e6b55c-5d96-4566-8f1a-c35a4097c57f-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-55hg57h\" (UID: \"61e6b55c-5d96-4566-8f1a-c35a4097c57f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-55hg57h"
May 26 11:52:44.672103 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:52:44.672082    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9qhj\" (UniqueName: \"kubernetes.io/projected/25b0394c-f00b-4f2e-acb4-853d2a617f79-kube-api-access-d9qhj\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7bc7c9f5cf8xqlh\" (UID: \"25b0394c-f00b-4f2e-acb4-853d2a617f79\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7bc7c9f5cf8xqlh"
May 26 11:52:44.673676 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:52:44.673649    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lf4rh\" (UniqueName: \"kubernetes.io/projected/61e6b55c-5d96-4566-8f1a-c35a4097c57f-kube-api-access-lf4rh\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-55hg57h\" (UID: \"61e6b55c-5d96-4566-8f1a-c35a4097c57f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-55hg57h"
May 26 11:52:44.676369 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:52:44.676347    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7bc7c9f5cf8xqlh"
May 26 11:52:44.698909 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:52:44.698883    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-55hg57h"
May 26 11:52:44.833650 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:52:44.833620    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7bc7c9f5cf8xqlh"]
May 26 11:52:44.837395 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:52:44.837367    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25b0394c_f00b_4f2e_acb4_853d2a617f79.slice/crio-87bdd5f9d207a5a2e144cf09ada35a75beb48f81bff1922e91137ed4f4f24121 WatchSource:0}: Error finding container 87bdd5f9d207a5a2e144cf09ada35a75beb48f81bff1922e91137ed4f4f24121: Status 404 returned error can't find the container with id 87bdd5f9d207a5a2e144cf09ada35a75beb48f81bff1922e91137ed4f4f24121
May 26 11:52:44.851433 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:52:44.851408    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-55hg57h"]
May 26 11:52:44.852367 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:52:44.852345    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61e6b55c_5d96_4566_8f1a_c35a4097c57f.slice/crio-1869cfaf891271fe504ab2f4571d443a1dc6bc79368ac99fcd86c37446fb1a35 WatchSource:0}: Error finding container 1869cfaf891271fe504ab2f4571d443a1dc6bc79368ac99fcd86c37446fb1a35: Status 404 returned error can't find the container with id 1869cfaf891271fe504ab2f4571d443a1dc6bc79368ac99fcd86c37446fb1a35
May 26 11:52:45.604333 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:52:45.604296    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7bc7c9f5cf8xqlh" event={"ID":"25b0394c-f00b-4f2e-acb4-853d2a617f79","Type":"ContainerStarted","Data":"87bdd5f9d207a5a2e144cf09ada35a75beb48f81bff1922e91137ed4f4f24121"}
May 26 11:52:45.605644 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:52:45.605620    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-55hg57h" event={"ID":"61e6b55c-5d96-4566-8f1a-c35a4097c57f","Type":"ContainerStarted","Data":"9843c7be8ce8e6f33efe657149ed2d6402e4771bdb359ae7cf5dd92ade20fa43"}
May 26 11:52:45.605713 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:52:45.605651    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-55hg57h" event={"ID":"61e6b55c-5d96-4566-8f1a-c35a4097c57f","Type":"ContainerStarted","Data":"1869cfaf891271fe504ab2f4571d443a1dc6bc79368ac99fcd86c37446fb1a35"}
May 26 11:52:46.614239 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:52:46.614179    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7bc7c9f5cf8xqlh" event={"ID":"25b0394c-f00b-4f2e-acb4-853d2a617f79","Type":"ContainerStarted","Data":"cea8c33e26f344652cff649ea777fcaa5050980811c9918470e2962549a5a5c5"}
May 26 11:52:46.614698 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:52:46.614342    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7bc7c9f5cf8xqlh"
May 26 11:52:47.619985 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:52:47.619945    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7bc7c9f5cf8xqlh" event={"ID":"25b0394c-f00b-4f2e-acb4-853d2a617f79","Type":"ContainerStarted","Data":"c5f89953ccd1141b95cd453fb99f697d69b23d9a6214a0955f57285aaa6536bb"}
May 26 11:52:50.632421 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:52:50.632384    2572 generic.go:358] "Generic (PLEG): container finished" podID="61e6b55c-5d96-4566-8f1a-c35a4097c57f" containerID="9843c7be8ce8e6f33efe657149ed2d6402e4771bdb359ae7cf5dd92ade20fa43" exitCode=0
May 26 11:52:50.632901 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:52:50.632456    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-55hg57h" event={"ID":"61e6b55c-5d96-4566-8f1a-c35a4097c57f","Type":"ContainerDied","Data":"9843c7be8ce8e6f33efe657149ed2d6402e4771bdb359ae7cf5dd92ade20fa43"}
May 26 11:52:51.637239 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:52:51.637180    2572 generic.go:358] "Generic (PLEG): container finished" podID="25b0394c-f00b-4f2e-acb4-853d2a617f79" containerID="c5f89953ccd1141b95cd453fb99f697d69b23d9a6214a0955f57285aaa6536bb" exitCode=0
May 26 11:52:51.637239 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:52:51.637229    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7bc7c9f5cf8xqlh" event={"ID":"25b0394c-f00b-4f2e-acb4-853d2a617f79","Type":"ContainerDied","Data":"c5f89953ccd1141b95cd453fb99f697d69b23d9a6214a0955f57285aaa6536bb"}
May 26 11:52:51.639004 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:52:51.638977    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-55hg57h" event={"ID":"61e6b55c-5d96-4566-8f1a-c35a4097c57f","Type":"ContainerStarted","Data":"3afff25ad5eaab36ab40dbf91e9903cc3eff98a1a6caa8d9e780cc4bd1c084cf"}
May 26 11:52:51.719616 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:52:51.719558    2572 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-55hg57h" podStartSLOduration=7.719538839 podStartE2EDuration="7.719538839s" podCreationTimestamp="2026-05-26 11:52:44 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 11:52:51.716956399 +0000 UTC m=+1694.619166728" watchObservedRunningTime="2026-05-26 11:52:51.719538839 +0000 UTC m=+1694.621749181"
May 26 11:52:52.648397 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:52:52.648357    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7bc7c9f5cf8xqlh" event={"ID":"25b0394c-f00b-4f2e-acb4-853d2a617f79","Type":"ContainerStarted","Data":"114d5cdc6116bd63a47e3ecee497cc7c9db5c2cd5f6075b4eaa8f37b22c3cbe7"}
May 26 11:52:52.672082 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:52:52.672008    2572 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7bc7c9f5cf8xqlh" podStartSLOduration=7.872373449 podStartE2EDuration="8.671993143s" podCreationTimestamp="2026-05-26 11:52:44 +0000 UTC" firstStartedPulling="2026-05-26 11:52:44.84178034 +0000 UTC m=+1687.743990647" lastFinishedPulling="2026-05-26 11:52:45.641400034 +0000 UTC m=+1688.543610341" observedRunningTime="2026-05-26 11:52:52.669966063 +0000 UTC m=+1695.572176393" watchObservedRunningTime="2026-05-26 11:52:52.671993143 +0000 UTC m=+1695.574203474"
May 26 11:52:54.676498 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:52:54.676399    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7bc7c9f5cf8xqlh"
May 26 11:52:54.676498 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:52:54.676446    2572 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7bc7c9f5cf8xqlh"
May 26 11:52:54.677937 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:52:54.677895    2572 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7bc7c9f5cf8xqlh" podUID="25b0394c-f00b-4f2e-acb4-853d2a617f79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8001/health\": dial tcp 10.132.0.45:8001: connect: connection refused"
May 26 11:52:54.699264 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:52:54.699226    2572 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-55hg57h"
May 26 11:52:54.699264 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:52:54.699265    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-55hg57h"
May 26 11:52:54.700653 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:52:54.700619    2572 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-55hg57h" podUID="61e6b55c-5d96-4566-8f1a-c35a4097c57f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/health\": dial tcp 10.132.0.46:8000: connect: connection refused"
May 26 11:53:04.416031 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:53:04.416008    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-test-kserve-76745cd4f8-t96vl_bcec67c7-b591-4630-b83d-1fffc2fd3dea/main/0.log"
May 26 11:53:04.416469 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:53:04.416450    2572 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-76745cd4f8-t96vl"
May 26 11:53:04.568042 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:53:04.568004    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bcec67c7-b591-4630-b83d-1fffc2fd3dea-tmp-dir\") pod \"bcec67c7-b591-4630-b83d-1fffc2fd3dea\" (UID: \"bcec67c7-b591-4630-b83d-1fffc2fd3dea\") "
May 26 11:53:04.568265 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:53:04.568058    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bcec67c7-b591-4630-b83d-1fffc2fd3dea-dshm\") pod \"bcec67c7-b591-4630-b83d-1fffc2fd3dea\" (UID: \"bcec67c7-b591-4630-b83d-1fffc2fd3dea\") "
May 26 11:53:04.568265 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:53:04.568105    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bcec67c7-b591-4630-b83d-1fffc2fd3dea-model-cache\") pod \"bcec67c7-b591-4630-b83d-1fffc2fd3dea\" (UID: \"bcec67c7-b591-4630-b83d-1fffc2fd3dea\") "
May 26 11:53:04.568265 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:53:04.568134    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bcec67c7-b591-4630-b83d-1fffc2fd3dea-kserve-provision-location\") pod \"bcec67c7-b591-4630-b83d-1fffc2fd3dea\" (UID: \"bcec67c7-b591-4630-b83d-1fffc2fd3dea\") "
May 26 11:53:04.568265 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:53:04.568165    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bcec67c7-b591-4630-b83d-1fffc2fd3dea-home\") pod \"bcec67c7-b591-4630-b83d-1fffc2fd3dea\" (UID: \"bcec67c7-b591-4630-b83d-1fffc2fd3dea\") "
May 26 11:53:04.568265 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:53:04.568208    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f6x9z\" (UniqueName: \"kubernetes.io/projected/bcec67c7-b591-4630-b83d-1fffc2fd3dea-kube-api-access-f6x9z\") pod \"bcec67c7-b591-4630-b83d-1fffc2fd3dea\" (UID: \"bcec67c7-b591-4630-b83d-1fffc2fd3dea\") "
May 26 11:53:04.568265 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:53:04.568250    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bcec67c7-b591-4630-b83d-1fffc2fd3dea-tls-certs\") pod \"bcec67c7-b591-4630-b83d-1fffc2fd3dea\" (UID: \"bcec67c7-b591-4630-b83d-1fffc2fd3dea\") "
May 26 11:53:04.568579 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:53:04.568388    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcec67c7-b591-4630-b83d-1fffc2fd3dea-model-cache" (OuterVolumeSpecName: "model-cache") pod "bcec67c7-b591-4630-b83d-1fffc2fd3dea" (UID: "bcec67c7-b591-4630-b83d-1fffc2fd3dea"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 11:53:04.568671 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:53:04.568629    2572 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bcec67c7-b591-4630-b83d-1fffc2fd3dea-model-cache\") on node \"ip-10-0-136-194.ec2.internal\" DevicePath \"\""
May 26 11:53:04.568791 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:53:04.568763    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcec67c7-b591-4630-b83d-1fffc2fd3dea-home" (OuterVolumeSpecName: "home") pod "bcec67c7-b591-4630-b83d-1fffc2fd3dea" (UID: "bcec67c7-b591-4630-b83d-1fffc2fd3dea"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 11:53:04.572036 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:53:04.571994    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcec67c7-b591-4630-b83d-1fffc2fd3dea-dshm" (OuterVolumeSpecName: "dshm") pod "bcec67c7-b591-4630-b83d-1fffc2fd3dea" (UID: "bcec67c7-b591-4630-b83d-1fffc2fd3dea"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 11:53:04.572356 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:53:04.572324    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bcec67c7-b591-4630-b83d-1fffc2fd3dea-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "bcec67c7-b591-4630-b83d-1fffc2fd3dea" (UID: "bcec67c7-b591-4630-b83d-1fffc2fd3dea"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 11:53:04.572584 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:53:04.572557    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bcec67c7-b591-4630-b83d-1fffc2fd3dea-kube-api-access-f6x9z" (OuterVolumeSpecName: "kube-api-access-f6x9z") pod "bcec67c7-b591-4630-b83d-1fffc2fd3dea" (UID: "bcec67c7-b591-4630-b83d-1fffc2fd3dea"). InnerVolumeSpecName "kube-api-access-f6x9z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 11:53:04.584690 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:53:04.584648    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcec67c7-b591-4630-b83d-1fffc2fd3dea-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "bcec67c7-b591-4630-b83d-1fffc2fd3dea" (UID: "bcec67c7-b591-4630-b83d-1fffc2fd3dea"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 11:53:04.620449 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:53:04.620394    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcec67c7-b591-4630-b83d-1fffc2fd3dea-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "bcec67c7-b591-4630-b83d-1fffc2fd3dea" (UID: "bcec67c7-b591-4630-b83d-1fffc2fd3dea"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 11:53:04.669763 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:53:04.669723    2572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bcec67c7-b591-4630-b83d-1fffc2fd3dea-tls-certs\") on node \"ip-10-0-136-194.ec2.internal\" DevicePath \"\""
May 26 11:53:04.669763 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:53:04.669762    2572 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bcec67c7-b591-4630-b83d-1fffc2fd3dea-tmp-dir\") on node \"ip-10-0-136-194.ec2.internal\" DevicePath \"\""
May 26 11:53:04.670225 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:53:04.669777    2572 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bcec67c7-b591-4630-b83d-1fffc2fd3dea-dshm\") on node \"ip-10-0-136-194.ec2.internal\" DevicePath \"\""
May 26 11:53:04.670225 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:53:04.669792    2572 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bcec67c7-b591-4630-b83d-1fffc2fd3dea-kserve-provision-location\") on node \"ip-10-0-136-194.ec2.internal\" DevicePath \"\""
May 26 11:53:04.670225 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:53:04.669807    2572 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bcec67c7-b591-4630-b83d-1fffc2fd3dea-home\") on node \"ip-10-0-136-194.ec2.internal\" DevicePath \"\""
May 26 11:53:04.670225 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:53:04.669822    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f6x9z\" (UniqueName: \"kubernetes.io/projected/bcec67c7-b591-4630-b83d-1fffc2fd3dea-kube-api-access-f6x9z\") on node \"ip-10-0-136-194.ec2.internal\" DevicePath \"\""
May 26 11:53:04.676817 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:53:04.676756    2572 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7bc7c9f5cf8xqlh" podUID="25b0394c-f00b-4f2e-acb4-853d2a617f79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8001/health\": dial tcp 10.132.0.45:8001: connect: connection refused"
May 26 11:53:04.696347 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:53:04.696321    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-test-kserve-76745cd4f8-t96vl_bcec67c7-b591-4630-b83d-1fffc2fd3dea/main/0.log"
May 26 11:53:04.696766 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:53:04.696728    2572 generic.go:358] "Generic (PLEG): container finished" podID="bcec67c7-b591-4630-b83d-1fffc2fd3dea" containerID="0e65dac81bc1976de0cb18a2c44fe5d203bcc4cbe41fb52f929ce79b2443b153" exitCode=137
May 26 11:53:04.696900 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:53:04.696774    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-76745cd4f8-t96vl" event={"ID":"bcec67c7-b591-4630-b83d-1fffc2fd3dea","Type":"ContainerDied","Data":"0e65dac81bc1976de0cb18a2c44fe5d203bcc4cbe41fb52f929ce79b2443b153"}
May 26 11:53:04.696900 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:53:04.696813    2572 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-76745cd4f8-t96vl"
May 26 11:53:04.696900 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:53:04.696819    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-76745cd4f8-t96vl" event={"ID":"bcec67c7-b591-4630-b83d-1fffc2fd3dea","Type":"ContainerDied","Data":"04a40586fd4fbf89d53733d26335f0a87d61a01a906117d81cf5b20223bfc16d"}
May 26 11:53:04.696900 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:53:04.696840    2572 scope.go:117] "RemoveContainer" containerID="0e65dac81bc1976de0cb18a2c44fe5d203bcc4cbe41fb52f929ce79b2443b153"
May 26 11:53:04.697708 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:53:04.697685    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7bc7c9f5cf8xqlh"
May 26 11:53:04.699725 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:53:04.699696    2572 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-55hg57h" podUID="61e6b55c-5d96-4566-8f1a-c35a4097c57f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/health\": dial tcp 10.132.0.46:8000: connect: connection refused"
May 26 11:53:04.706832 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:53:04.706810    2572 scope.go:117] "RemoveContainer" containerID="0197db812af5d2361920da7a0bbbf37d0767f588740349342270d1feae5733c3"
May 26 11:53:04.723558 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:53:04.723539    2572 scope.go:117] "RemoveContainer" containerID="0e65dac81bc1976de0cb18a2c44fe5d203bcc4cbe41fb52f929ce79b2443b153"
May 26 11:53:04.723862 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:53:04.723829    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e65dac81bc1976de0cb18a2c44fe5d203bcc4cbe41fb52f929ce79b2443b153\": container with ID starting with 0e65dac81bc1976de0cb18a2c44fe5d203bcc4cbe41fb52f929ce79b2443b153 not found: ID does not exist" containerID="0e65dac81bc1976de0cb18a2c44fe5d203bcc4cbe41fb52f929ce79b2443b153"
May 26 11:53:04.723960 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:53:04.723867    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e65dac81bc1976de0cb18a2c44fe5d203bcc4cbe41fb52f929ce79b2443b153"} err="failed to get container status \"0e65dac81bc1976de0cb18a2c44fe5d203bcc4cbe41fb52f929ce79b2443b153\": rpc error: code = NotFound desc = could not find container \"0e65dac81bc1976de0cb18a2c44fe5d203bcc4cbe41fb52f929ce79b2443b153\": container with ID starting with 0e65dac81bc1976de0cb18a2c44fe5d203bcc4cbe41fb52f929ce79b2443b153 not found: ID does not exist"
May 26 11:53:04.723960 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:53:04.723891    2572 scope.go:117] "RemoveContainer" containerID="0197db812af5d2361920da7a0bbbf37d0767f588740349342270d1feae5733c3"
May 26 11:53:04.724168 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:53:04.724148    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0197db812af5d2361920da7a0bbbf37d0767f588740349342270d1feae5733c3\": container with ID starting with 0197db812af5d2361920da7a0bbbf37d0767f588740349342270d1feae5733c3 not found: ID does not exist" containerID="0197db812af5d2361920da7a0bbbf37d0767f588740349342270d1feae5733c3"
May 26 11:53:04.724289 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:53:04.724173    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0197db812af5d2361920da7a0bbbf37d0767f588740349342270d1feae5733c3"} err="failed to get container status \"0197db812af5d2361920da7a0bbbf37d0767f588740349342270d1feae5733c3\": rpc error: code = NotFound desc = could not find container \"0197db812af5d2361920da7a0bbbf37d0767f588740349342270d1feae5733c3\": container with ID starting with 0197db812af5d2361920da7a0bbbf37d0767f588740349342270d1feae5733c3 not found: ID does not exist"
May 26 11:53:04.734816 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:53:04.734790    2572 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-76745cd4f8-t96vl"]
May 26 11:53:04.738122 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:53:04.738096    2572 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-76745cd4f8-t96vl"]
May 26 11:53:05.678279 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:53:05.678237    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bcec67c7-b591-4630-b83d-1fffc2fd3dea" path="/var/lib/kubelet/pods/bcec67c7-b591-4630-b83d-1fffc2fd3dea/volumes"
May 26 11:53:14.677598 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:53:14.677548    2572 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7bc7c9f5cf8xqlh" podUID="25b0394c-f00b-4f2e-acb4-853d2a617f79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8001/health\": dial tcp 10.132.0.45:8001: connect: connection refused"
May 26 11:53:14.699289 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:53:14.699242    2572 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-55hg57h" podUID="61e6b55c-5d96-4566-8f1a-c35a4097c57f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/health\": dial tcp 10.132.0.46:8000: connect: connection refused"
May 26 11:53:24.677764 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:53:24.677712    2572 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7bc7c9f5cf8xqlh" podUID="25b0394c-f00b-4f2e-acb4-853d2a617f79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8001/health\": dial tcp 10.132.0.45:8001: connect: connection refused"
May 26 11:53:24.700181 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:53:24.700123    2572 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-55hg57h" podUID="61e6b55c-5d96-4566-8f1a-c35a4097c57f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/health\": dial tcp 10.132.0.46:8000: connect: connection refused"
May 26 11:53:34.677584 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:53:34.677533    2572 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7bc7c9f5cf8xqlh" podUID="25b0394c-f00b-4f2e-acb4-853d2a617f79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8001/health\": dial tcp 10.132.0.45:8001: connect: connection refused"
May 26 11:53:34.699349 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:53:34.699313    2572 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-55hg57h" podUID="61e6b55c-5d96-4566-8f1a-c35a4097c57f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/health\": dial tcp 10.132.0.46:8000: connect: connection refused"
May 26 11:53:44.677255 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:53:44.677205    2572 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7bc7c9f5cf8xqlh" podUID="25b0394c-f00b-4f2e-acb4-853d2a617f79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8001/health\": dial tcp 10.132.0.45:8001: connect: connection refused"
May 26 11:53:44.699519 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:53:44.699482    2572 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-55hg57h" podUID="61e6b55c-5d96-4566-8f1a-c35a4097c57f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/health\": dial tcp 10.132.0.46:8000: connect: connection refused"
May 26 11:53:54.676749 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:53:54.676691    2572 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7bc7c9f5cf8xqlh" podUID="25b0394c-f00b-4f2e-acb4-853d2a617f79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8001/health\": dial tcp 10.132.0.45:8001: connect: connection refused"
May 26 11:53:54.699303 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:53:54.699272    2572 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-55hg57h" podUID="61e6b55c-5d96-4566-8f1a-c35a4097c57f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/health\": dial tcp 10.132.0.46:8000: connect: connection refused"
May 26 11:54:04.677783 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:54:04.677732    2572 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7bc7c9f5cf8xqlh" podUID="25b0394c-f00b-4f2e-acb4-853d2a617f79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8001/health\": dial tcp 10.132.0.45:8001: connect: connection refused"
May 26 11:54:04.699381 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:54:04.699346    2572 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-55hg57h" podUID="61e6b55c-5d96-4566-8f1a-c35a4097c57f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/health\": dial tcp 10.132.0.46:8000: connect: connection refused"
May 26 11:54:14.677441 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:54:14.677387    2572 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7bc7c9f5cf8xqlh" podUID="25b0394c-f00b-4f2e-acb4-853d2a617f79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8001/health\": dial tcp 10.132.0.45:8001: connect: connection refused"
May 26 11:54:14.699274 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:54:14.699245    2572 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-55hg57h" podUID="61e6b55c-5d96-4566-8f1a-c35a4097c57f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/health\": dial tcp 10.132.0.46:8000: connect: connection refused"
May 26 11:54:24.677345 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:54:24.677236    2572 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7bc7c9f5cf8xqlh" podUID="25b0394c-f00b-4f2e-acb4-853d2a617f79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8001/health\": dial tcp 10.132.0.45:8001: connect: connection refused"
May 26 11:54:24.699446 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:54:24.699410    2572 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-55hg57h" podUID="61e6b55c-5d96-4566-8f1a-c35a4097c57f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/health\": dial tcp 10.132.0.46:8000: connect: connection refused"
May 26 11:54:34.677537 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:54:34.677488    2572 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7bc7c9f5cf8xqlh" podUID="25b0394c-f00b-4f2e-acb4-853d2a617f79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8001/health\": dial tcp 10.132.0.45:8001: connect: connection refused"
May 26 11:54:34.699980 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:54:34.699942    2572 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-55hg57h" podUID="61e6b55c-5d96-4566-8f1a-c35a4097c57f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/health\": dial tcp 10.132.0.46:8000: connect: connection refused"
May 26 11:54:37.710215 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:54:37.710157    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-5bfgg_c6373342-dd07-4497-a4f3-978ea9613b6e/console-operator/2.log"
May 26 11:54:37.723742 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:54:37.723712    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-5bfgg_c6373342-dd07-4497-a4f3-978ea9613b6e/console-operator/2.log"
May 26 11:54:37.724884 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:54:37.724857    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-js8cc_7264c907-f536-4719-905a-2bbda9e3c336/ovn-acl-logging/0.log"
May 26 11:54:37.730427 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:54:37.730405    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-js8cc_7264c907-f536-4719-905a-2bbda9e3c336/ovn-acl-logging/0.log"
May 26 11:54:44.677585 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:54:44.677537    2572 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7bc7c9f5cf8xqlh" podUID="25b0394c-f00b-4f2e-acb4-853d2a617f79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8001/health\": dial tcp 10.132.0.45:8001: connect: connection refused"
May 26 11:54:44.699386 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:54:44.699348    2572 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-55hg57h" podUID="61e6b55c-5d96-4566-8f1a-c35a4097c57f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/health\": dial tcp 10.132.0.46:8000: connect: connection refused"
May 26 11:54:54.676979 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:54:54.676930    2572 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7bc7c9f5cf8xqlh" podUID="25b0394c-f00b-4f2e-acb4-853d2a617f79" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8001/health\": dial tcp 10.132.0.45:8001: connect: connection refused"
May 26 11:54:54.699692 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:54:54.699661    2572 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-55hg57h" podUID="61e6b55c-5d96-4566-8f1a-c35a4097c57f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/health\": dial tcp 10.132.0.46:8000: connect: connection refused"
May 26 11:55:04.686463 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:55:04.686423    2572 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7bc7c9f5cf8xqlh"
May 26 11:55:04.698222 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:55:04.698181    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7bc7c9f5cf8xqlh"
May 26 11:55:04.714880 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:55:04.714851    2572 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-55hg57h"
May 26 11:55:04.724816 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:55:04.724784    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-55hg57h"
May 26 11:55:12.998917 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:55:12.998884    2572 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7bc7c9f5cf8xqlh"]
May 26 11:55:12.999365 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:55:12.999275    2572 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7bc7c9f5cf8xqlh" podUID="25b0394c-f00b-4f2e-acb4-853d2a617f79" containerName="main" containerID="cri-o://114d5cdc6116bd63a47e3ecee497cc7c9db5c2cd5f6075b4eaa8f37b22c3cbe7" gracePeriod=30
May 26 11:55:13.002486 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:55:13.002460    2572 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-55hg57h"]
May 26 11:55:13.002799 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:55:13.002752    2572 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-55hg57h" podUID="61e6b55c-5d96-4566-8f1a-c35a4097c57f" containerName="main" containerID="cri-o://3afff25ad5eaab36ab40dbf91e9903cc3eff98a1a6caa8d9e780cc4bd1c084cf" gracePeriod=30
May 26 11:55:22.182122 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:55:22.182088    2572 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7955f5555b-4jdzr"]
May 26 11:55:22.182701 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:55:22.182686    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bcec67c7-b591-4630-b83d-1fffc2fd3dea" containerName="storage-initializer"
May 26 11:55:22.182773 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:55:22.182707    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="bcec67c7-b591-4630-b83d-1fffc2fd3dea" containerName="storage-initializer"
May 26 11:55:22.182773 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:55:22.182723    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bcec67c7-b591-4630-b83d-1fffc2fd3dea" containerName="main"
May 26 11:55:22.182773 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:55:22.182732    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="bcec67c7-b591-4630-b83d-1fffc2fd3dea" containerName="main"
May 26 11:55:22.182930 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:55:22.182867    2572 memory_manager.go:356] "RemoveStaleState removing state" podUID="bcec67c7-b591-4630-b83d-1fffc2fd3dea" containerName="main"
May 26 11:55:22.187843 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:55:22.187820    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7955f5555b-4jdzr"
May 26 11:55:22.190721 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:55:22.190700    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"custom-route-timeout-pd-test-kserve-self-signed-certs\""
May 26 11:55:22.190839 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:55:22.190774    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"custom-route-timeout-pd-test-kserve-dockercfg-hz7l5\""
May 26 11:55:22.191944 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:55:22.191921    2572 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-8578d996bc-4f4n5"]
May 26 11:55:22.197256 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:55:22.197236    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-8578d996bc-4f4n5"
May 26 11:55:22.199143 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:55:22.199121    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7955f5555b-4jdzr"]
May 26 11:55:22.213845 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:55:22.213822    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-8578d996bc-4f4n5"]
May 26 11:55:22.301917 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:55:22.301880    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9345be29-4f25-41c5-8489-c02aa4b7cea6-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-7955f5555b-4jdzr\" (UID: \"9345be29-4f25-41c5-8489-c02aa4b7cea6\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7955f5555b-4jdzr"
May 26 11:55:22.301917 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:55:22.301919    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9345be29-4f25-41c5-8489-c02aa4b7cea6-home\") pod \"custom-route-timeout-pd-test-kserve-7955f5555b-4jdzr\" (UID: \"9345be29-4f25-41c5-8489-c02aa4b7cea6\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7955f5555b-4jdzr"
May 26 11:55:22.302127 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:55:22.301963    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f0068a6-2aad-417c-a347-14a7cbcc4bf5-home\") pod \"custom-route-timeout-pd-test-kserve-prefill-8578d996bc-4f4n5\" (UID: \"8f0068a6-2aad-417c-a347-14a7cbcc4bf5\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-8578d996bc-4f4n5"
May 26 11:55:22.302127 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:55:22.301982    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8q865\" (UniqueName: \"kubernetes.io/projected/9345be29-4f25-41c5-8489-c02aa4b7cea6-kube-api-access-8q865\") pod \"custom-route-timeout-pd-test-kserve-7955f5555b-4jdzr\" (UID: \"9345be29-4f25-41c5-8489-c02aa4b7cea6\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7955f5555b-4jdzr"
May 26 11:55:22.302127 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:55:22.302024    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f0068a6-2aad-417c-a347-14a7cbcc4bf5-model-cache\") pod \"custom-route-timeout-pd-test-kserve-prefill-8578d996bc-4f4n5\" (UID: \"8f0068a6-2aad-417c-a347-14a7cbcc4bf5\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-8578d996bc-4f4n5"
May 26 11:55:22.302127 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:55:22.302042    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvfjc\" (UniqueName: \"kubernetes.io/projected/8f0068a6-2aad-417c-a347-14a7cbcc4bf5-kube-api-access-vvfjc\") pod \"custom-route-timeout-pd-test-kserve-prefill-8578d996bc-4f4n5\" (UID: \"8f0068a6-2aad-417c-a347-14a7cbcc4bf5\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-8578d996bc-4f4n5"
May 26 11:55:22.302127 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:55:22.302093    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9345be29-4f25-41c5-8489-c02aa4b7cea6-model-cache\") pod \"custom-route-timeout-pd-test-kserve-7955f5555b-4jdzr\" (UID: \"9345be29-4f25-41c5-8489-c02aa4b7cea6\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7955f5555b-4jdzr"
May 26 11:55:22.302127 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:55:22.302117    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9345be29-4f25-41c5-8489-c02aa4b7cea6-dshm\") pod \"custom-route-timeout-pd-test-kserve-7955f5555b-4jdzr\" (UID: \"9345be29-4f25-41c5-8489-c02aa4b7cea6\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7955f5555b-4jdzr"
May 26 11:55:22.302351 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:55:22.302133    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8f0068a6-2aad-417c-a347-14a7cbcc4bf5-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-prefill-8578d996bc-4f4n5\" (UID: \"8f0068a6-2aad-417c-a347-14a7cbcc4bf5\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-8578d996bc-4f4n5"
May 26 11:55:22.302351 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:55:22.302157    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9345be29-4f25-41c5-8489-c02aa4b7cea6-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-7955f5555b-4jdzr\" (UID: \"9345be29-4f25-41c5-8489-c02aa4b7cea6\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7955f5555b-4jdzr"
May 26 11:55:22.302351 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:55:22.302227    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f0068a6-2aad-417c-a347-14a7cbcc4bf5-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-prefill-8578d996bc-4f4n5\" (UID: \"8f0068a6-2aad-417c-a347-14a7cbcc4bf5\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-8578d996bc-4f4n5"
May 26 11:55:22.302351 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:55:22.302245    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f0068a6-2aad-417c-a347-14a7cbcc4bf5-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-prefill-8578d996bc-4f4n5\" (UID: \"8f0068a6-2aad-417c-a347-14a7cbcc4bf5\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-8578d996bc-4f4n5"
May 26 11:55:22.302351 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:55:22.302258    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9345be29-4f25-41c5-8489-c02aa4b7cea6-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-7955f5555b-4jdzr\" (UID: \"9345be29-4f25-41c5-8489-c02aa4b7cea6\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7955f5555b-4jdzr"
May 26 11:55:22.302351 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:55:22.302299    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f0068a6-2aad-417c-a347-14a7cbcc4bf5-dshm\") pod \"custom-route-timeout-pd-test-kserve-prefill-8578d996bc-4f4n5\" (UID: \"8f0068a6-2aad-417c-a347-14a7cbcc4bf5\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-8578d996bc-4f4n5"
May 26 11:55:22.403532 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:55:22.403505    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9345be29-4f25-41c5-8489-c02aa4b7cea6-model-cache\") pod \"custom-route-timeout-pd-test-kserve-7955f5555b-4jdzr\" (UID: \"9345be29-4f25-41c5-8489-c02aa4b7cea6\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7955f5555b-4jdzr"
May 26 11:55:22.410653 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:55:22.403545    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9345be29-4f25-41c5-8489-c02aa4b7cea6-dshm\") pod \"custom-route-timeout-pd-test-kserve-7955f5555b-4jdzr\" (UID: \"9345be29-4f25-41c5-8489-c02aa4b7cea6\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7955f5555b-4jdzr"
May 26 11:55:22.410653 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:55:22.403564    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8f0068a6-2aad-417c-a347-14a7cbcc4bf5-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-prefill-8578d996bc-4f4n5\" (UID: \"8f0068a6-2aad-417c-a347-14a7cbcc4bf5\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-8578d996bc-4f4n5"
May 26 11:55:22.410653 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:55:22.403584    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9345be29-4f25-41c5-8489-c02aa4b7cea6-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-7955f5555b-4jdzr\" (UID: \"9345be29-4f25-41c5-8489-c02aa4b7cea6\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7955f5555b-4jdzr"
May 26 11:55:22.410653 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:55:22.403620    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f0068a6-2aad-417c-a347-14a7cbcc4bf5-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-prefill-8578d996bc-4f4n5\" (UID: \"8f0068a6-2aad-417c-a347-14a7cbcc4bf5\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-8578d996bc-4f4n5"
May 26 11:55:22.410653 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:55:22.403642    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f0068a6-2aad-417c-a347-14a7cbcc4bf5-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-prefill-8578d996bc-4f4n5\" (UID: \"8f0068a6-2aad-417c-a347-14a7cbcc4bf5\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-8578d996bc-4f4n5"
May 26 11:55:22.410653 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:55:22.403659    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9345be29-4f25-41c5-8489-c02aa4b7cea6-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-7955f5555b-4jdzr\" (UID: \"9345be29-4f25-41c5-8489-c02aa4b7cea6\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7955f5555b-4jdzr"
May 26 11:55:22.410653 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:55:22.403684    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f0068a6-2aad-417c-a347-14a7cbcc4bf5-dshm\") pod \"custom-route-timeout-pd-test-kserve-prefill-8578d996bc-4f4n5\" (UID: \"8f0068a6-2aad-417c-a347-14a7cbcc4bf5\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-8578d996bc-4f4n5"
May 26 11:55:22.410653 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:55:22.403719    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9345be29-4f25-41c5-8489-c02aa4b7cea6-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-7955f5555b-4jdzr\" (UID: \"9345be29-4f25-41c5-8489-c02aa4b7cea6\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7955f5555b-4jdzr"
May 26 11:55:22.410653 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:55:22.403778    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9345be29-4f25-41c5-8489-c02aa4b7cea6-home\") pod \"custom-route-timeout-pd-test-kserve-7955f5555b-4jdzr\" (UID: \"9345be29-4f25-41c5-8489-c02aa4b7cea6\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7955f5555b-4jdzr"
May 26 11:55:22.410653 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:55:22.403849    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f0068a6-2aad-417c-a347-14a7cbcc4bf5-home\") pod \"custom-route-timeout-pd-test-kserve-prefill-8578d996bc-4f4n5\" (UID: \"8f0068a6-2aad-417c-a347-14a7cbcc4bf5\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-8578d996bc-4f4n5"
May 26 11:55:22.410653 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:55:22.403880    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-8q865\" (UniqueName: \"kubernetes.io/projected/9345be29-4f25-41c5-8489-c02aa4b7cea6-kube-api-access-8q865\") pod \"custom-route-timeout-pd-test-kserve-7955f5555b-4jdzr\" (UID: \"9345be29-4f25-41c5-8489-c02aa4b7cea6\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7955f5555b-4jdzr"
May 26 11:55:22.410653 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:55:22.403919    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f0068a6-2aad-417c-a347-14a7cbcc4bf5-model-cache\") pod \"custom-route-timeout-pd-test-kserve-prefill-8578d996bc-4f4n5\" (UID: \"8f0068a6-2aad-417c-a347-14a7cbcc4bf5\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-8578d996bc-4f4n5"
May 26 11:55:22.410653 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:55:22.403947    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-vvfjc\" (UniqueName: \"kubernetes.io/projected/8f0068a6-2aad-417c-a347-14a7cbcc4bf5-kube-api-access-vvfjc\") pod \"custom-route-timeout-pd-test-kserve-prefill-8578d996bc-4f4n5\" (UID: \"8f0068a6-2aad-417c-a347-14a7cbcc4bf5\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-8578d996bc-4f4n5"
May 26 11:55:22.410653 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:55:22.404004    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8f0068a6-2aad-417c-a347-14a7cbcc4bf5-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-prefill-8578d996bc-4f4n5\" (UID: \"8f0068a6-2aad-417c-a347-14a7cbcc4bf5\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-8578d996bc-4f4n5"
May 26 11:55:22.411166 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:55:22.404046    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9345be29-4f25-41c5-8489-c02aa4b7cea6-model-cache\") pod \"custom-route-timeout-pd-test-kserve-7955f5555b-4jdzr\" (UID: \"9345be29-4f25-41c5-8489-c02aa4b7cea6\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7955f5555b-4jdzr"
May 26 11:55:22.411166 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:55:22.404226    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9345be29-4f25-41c5-8489-c02aa4b7cea6-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-7955f5555b-4jdzr\" (UID: \"9345be29-4f25-41c5-8489-c02aa4b7cea6\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7955f5555b-4jdzr"
May 26 11:55:22.411166 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:55:22.404274    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f0068a6-2aad-417c-a347-14a7cbcc4bf5-home\") pod \"custom-route-timeout-pd-test-kserve-prefill-8578d996bc-4f4n5\" (UID: \"8f0068a6-2aad-417c-a347-14a7cbcc4bf5\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-8578d996bc-4f4n5"
May 26 11:55:22.411166 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:55:22.404517    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9345be29-4f25-41c5-8489-c02aa4b7cea6-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-7955f5555b-4jdzr\" (UID: \"9345be29-4f25-41c5-8489-c02aa4b7cea6\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7955f5555b-4jdzr"
May 26 11:55:22.411166 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:55:22.404563    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9345be29-4f25-41c5-8489-c02aa4b7cea6-home\") pod \"custom-route-timeout-pd-test-kserve-7955f5555b-4jdzr\" (UID: \"9345be29-4f25-41c5-8489-c02aa4b7cea6\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7955f5555b-4jdzr"
May 26 11:55:22.411166 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:55:22.404736    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f0068a6-2aad-417c-a347-14a7cbcc4bf5-model-cache\") pod \"custom-route-timeout-pd-test-kserve-prefill-8578d996bc-4f4n5\" (UID: \"8f0068a6-2aad-417c-a347-14a7cbcc4bf5\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-8578d996bc-4f4n5"
May 26 11:55:22.411166 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:55:22.404809    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f0068a6-2aad-417c-a347-14a7cbcc4bf5-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-prefill-8578d996bc-4f4n5\" (UID: \"8f0068a6-2aad-417c-a347-14a7cbcc4bf5\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-8578d996bc-4f4n5"
May 26 11:55:22.411166 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:55:22.406023    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f0068a6-2aad-417c-a347-14a7cbcc4bf5-dshm\") pod \"custom-route-timeout-pd-test-kserve-prefill-8578d996bc-4f4n5\" (UID: \"8f0068a6-2aad-417c-a347-14a7cbcc4bf5\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-8578d996bc-4f4n5"
May 26 11:55:22.411166 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:55:22.406087    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9345be29-4f25-41c5-8489-c02aa4b7cea6-dshm\") pod \"custom-route-timeout-pd-test-kserve-7955f5555b-4jdzr\" (UID: \"9345be29-4f25-41c5-8489-c02aa4b7cea6\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7955f5555b-4jdzr"
May 26 11:55:22.411166 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:55:22.406561    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f0068a6-2aad-417c-a347-14a7cbcc4bf5-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-prefill-8578d996bc-4f4n5\" (UID: \"8f0068a6-2aad-417c-a347-14a7cbcc4bf5\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-8578d996bc-4f4n5"
May 26 11:55:22.411166 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:55:22.406581    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9345be29-4f25-41c5-8489-c02aa4b7cea6-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-7955f5555b-4jdzr\" (UID: \"9345be29-4f25-41c5-8489-c02aa4b7cea6\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7955f5555b-4jdzr"
May 26 11:55:22.412646 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:55:22.412626    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvfjc\" (UniqueName: \"kubernetes.io/projected/8f0068a6-2aad-417c-a347-14a7cbcc4bf5-kube-api-access-vvfjc\") pod \"custom-route-timeout-pd-test-kserve-prefill-8578d996bc-4f4n5\" (UID: \"8f0068a6-2aad-417c-a347-14a7cbcc4bf5\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-8578d996bc-4f4n5"
May 26 11:55:22.413170 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:55:22.413151    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8q865\" (UniqueName: \"kubernetes.io/projected/9345be29-4f25-41c5-8489-c02aa4b7cea6-kube-api-access-8q865\") pod \"custom-route-timeout-pd-test-kserve-7955f5555b-4jdzr\" (UID: \"9345be29-4f25-41c5-8489-c02aa4b7cea6\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7955f5555b-4jdzr"
May 26 11:55:22.499698 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:55:22.499613    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7955f5555b-4jdzr"
May 26 11:55:22.508701 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:55:22.508671    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-8578d996bc-4f4n5"
May 26 11:55:22.648136 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:55:22.648101    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7955f5555b-4jdzr"]
May 26 11:55:22.652125 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:55:22.652093    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9345be29_4f25_41c5_8489_c02aa4b7cea6.slice/crio-983251a9cde037b28e39326ec2593a0f10b245f33566e05f51415c7b805efdcf WatchSource:0}: Error finding container 983251a9cde037b28e39326ec2593a0f10b245f33566e05f51415c7b805efdcf: Status 404 returned error can't find the container with id 983251a9cde037b28e39326ec2593a0f10b245f33566e05f51415c7b805efdcf
May 26 11:55:22.668065 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:55:22.668035    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-8578d996bc-4f4n5"]
May 26 11:55:22.672840 ip-10-0-136-194 kubenswrapper[2572]: W0526 11:55:22.672814    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f0068a6_2aad_417c_a347_14a7cbcc4bf5.slice/crio-139c7b87f888c240506f06c1e95a09e35aa9335602f85aaa2dff2d7f93205e6b WatchSource:0}: Error finding container 139c7b87f888c240506f06c1e95a09e35aa9335602f85aaa2dff2d7f93205e6b: Status 404 returned error can't find the container with id 139c7b87f888c240506f06c1e95a09e35aa9335602f85aaa2dff2d7f93205e6b
May 26 11:55:23.220059 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:55:23.220008    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-8578d996bc-4f4n5" event={"ID":"8f0068a6-2aad-417c-a347-14a7cbcc4bf5","Type":"ContainerStarted","Data":"8cc6b6b19864d9c1319126c6ad3a317501d1307610deac6bbe4de45eed4b5ace"}
May 26 11:55:23.220059 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:55:23.220058    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-8578d996bc-4f4n5" event={"ID":"8f0068a6-2aad-417c-a347-14a7cbcc4bf5","Type":"ContainerStarted","Data":"139c7b87f888c240506f06c1e95a09e35aa9335602f85aaa2dff2d7f93205e6b"}
May 26 11:55:23.221414 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:55:23.221382    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7955f5555b-4jdzr" event={"ID":"9345be29-4f25-41c5-8489-c02aa4b7cea6","Type":"ContainerStarted","Data":"6b7575685b8f04d1ea4f8b73617a67d16d911c9eb98929e3f5827bfa1b255dfe"}
May 26 11:55:23.221533 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:55:23.221420    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7955f5555b-4jdzr" event={"ID":"9345be29-4f25-41c5-8489-c02aa4b7cea6","Type":"ContainerStarted","Data":"983251a9cde037b28e39326ec2593a0f10b245f33566e05f51415c7b805efdcf"}
May 26 11:55:23.221533 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:55:23.221485    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7955f5555b-4jdzr"
May 26 11:55:24.228643 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:55:24.228606    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7955f5555b-4jdzr" event={"ID":"9345be29-4f25-41c5-8489-c02aa4b7cea6","Type":"ContainerStarted","Data":"7bf1b026f2a24314b13051d8e3b49573861946c826270ea57ba842dde540ec84"}
May 26 11:55:29.248925 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:55:29.248888    2572 generic.go:358] "Generic (PLEG): container finished" podID="9345be29-4f25-41c5-8489-c02aa4b7cea6" containerID="7bf1b026f2a24314b13051d8e3b49573861946c826270ea57ba842dde540ec84" exitCode=0
May 26 11:55:29.249350 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:55:29.248938    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7955f5555b-4jdzr" event={"ID":"9345be29-4f25-41c5-8489-c02aa4b7cea6","Type":"ContainerDied","Data":"7bf1b026f2a24314b13051d8e3b49573861946c826270ea57ba842dde540ec84"}
May 26 11:55:30.255779 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:55:30.255744    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7955f5555b-4jdzr" event={"ID":"9345be29-4f25-41c5-8489-c02aa4b7cea6","Type":"ContainerStarted","Data":"04fc740e951e25f21e8b7cfa3f8c9cdad7a5d6245627a0f74594cb14297fe6b5"}
May 26 11:55:30.295060 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:55:30.294997    2572 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7955f5555b-4jdzr" podStartSLOduration=8.294976779 podStartE2EDuration="8.294976779s" podCreationTimestamp="2026-05-26 11:55:22 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 11:55:30.293889572 +0000 UTC m=+1853.196099902" watchObservedRunningTime="2026-05-26 11:55:30.294976779 +0000 UTC m=+1853.197187109"
May 26 11:55:32.500228 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:55:32.500163    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7955f5555b-4jdzr"
May 26 11:55:32.500687 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:55:32.500274    2572 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7955f5555b-4jdzr"
May 26 11:55:32.501484 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:55:32.501437    2572 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7955f5555b-4jdzr" podUID="9345be29-4f25-41c5-8489-c02aa4b7cea6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8001/health\": dial tcp 10.132.0.47:8001: connect: connection refused"
May 26 11:55:32.513328 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:55:32.513303    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7955f5555b-4jdzr"
May 26 11:55:42.500627 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:55:42.500580    2572 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7955f5555b-4jdzr" podUID="9345be29-4f25-41c5-8489-c02aa4b7cea6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8001/health\": dial tcp 10.132.0.47:8001: connect: connection refused"
May 26 11:55:42.999830 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:55:42.999777    2572 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7bc7c9f5cf8xqlh" podUID="25b0394c-f00b-4f2e-acb4-853d2a617f79" containerName="llm-d-routing-sidecar" containerID="cri-o://cea8c33e26f344652cff649ea777fcaa5050980811c9918470e2962549a5a5c5" gracePeriod=2
May 26 11:55:43.313924 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:55:43.313897    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7bc7c9f5cf8xqlh_25b0394c-f00b-4f2e-acb4-853d2a617f79/main/0.log"
May 26 11:55:43.314613 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:55:43.314592    2572 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7bc7c9f5cf8xqlh"
May 26 11:55:43.315763 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:55:43.315746    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7bc7c9f5cf8xqlh_25b0394c-f00b-4f2e-acb4-853d2a617f79/main/0.log"
May 26 11:55:43.316416 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:55:43.316393    2572 generic.go:358] "Generic (PLEG): container finished" podID="25b0394c-f00b-4f2e-acb4-853d2a617f79" containerID="114d5cdc6116bd63a47e3ecee497cc7c9db5c2cd5f6075b4eaa8f37b22c3cbe7" exitCode=137
May 26 11:55:43.316416 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:55:43.316414    2572 generic.go:358] "Generic (PLEG): container finished" podID="25b0394c-f00b-4f2e-acb4-853d2a617f79" containerID="cea8c33e26f344652cff649ea777fcaa5050980811c9918470e2962549a5a5c5" exitCode=0
May 26 11:55:43.316545 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:55:43.316439    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7bc7c9f5cf8xqlh" event={"ID":"25b0394c-f00b-4f2e-acb4-853d2a617f79","Type":"ContainerDied","Data":"114d5cdc6116bd63a47e3ecee497cc7c9db5c2cd5f6075b4eaa8f37b22c3cbe7"}
May 26 11:55:43.316545 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:55:43.316475    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7bc7c9f5cf8xqlh" event={"ID":"25b0394c-f00b-4f2e-acb4-853d2a617f79","Type":"ContainerDied","Data":"cea8c33e26f344652cff649ea777fcaa5050980811c9918470e2962549a5a5c5"}
May 26 11:55:43.316545 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:55:43.316491    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7bc7c9f5cf8xqlh" event={"ID":"25b0394c-f00b-4f2e-acb4-853d2a617f79","Type":"ContainerDied","Data":"87bdd5f9d207a5a2e144cf09ada35a75beb48f81bff1922e91137ed4f4f24121"}
May 26 11:55:43.316545 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:55:43.316514    2572 scope.go:117] "RemoveContainer" containerID="114d5cdc6116bd63a47e3ecee497cc7c9db5c2cd5f6075b4eaa8f37b22c3cbe7"
May 26 11:55:43.318050 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:55:43.318026    2572 generic.go:358] "Generic (PLEG): container finished" podID="61e6b55c-5d96-4566-8f1a-c35a4097c57f" containerID="3afff25ad5eaab36ab40dbf91e9903cc3eff98a1a6caa8d9e780cc4bd1c084cf" exitCode=137
May 26 11:55:43.318145 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:55:43.318053    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-55hg57h" event={"ID":"61e6b55c-5d96-4566-8f1a-c35a4097c57f","Type":"ContainerDied","Data":"3afff25ad5eaab36ab40dbf91e9903cc3eff98a1a6caa8d9e780cc4bd1c084cf"}
May 26 11:55:43.318145 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:55:43.318087    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-55hg57h" event={"ID":"61e6b55c-5d96-4566-8f1a-c35a4097c57f","Type":"ContainerDied","Data":"1869cfaf891271fe504ab2f4571d443a1dc6bc79368ac99fcd86c37446fb1a35"}
May 26 11:55:43.318145 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:55:43.318102    2572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1869cfaf891271fe504ab2f4571d443a1dc6bc79368ac99fcd86c37446fb1a35"
May 26 11:55:43.319036 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:55:43.319021    2572 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-55hg57h"
May 26 11:55:43.329107 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:55:43.329089    2572 scope.go:117] "RemoveContainer" containerID="c5f89953ccd1141b95cd453fb99f697d69b23d9a6214a0955f57285aaa6536bb"
May 26 11:55:43.360667 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:55:43.360427    2572 scope.go:117] "RemoveContainer" containerID="cea8c33e26f344652cff649ea777fcaa5050980811c9918470e2962549a5a5c5"
May 26 11:55:43.371556 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:55:43.371536    2572 scope.go:117] "RemoveContainer" containerID="114d5cdc6116bd63a47e3ecee497cc7c9db5c2cd5f6075b4eaa8f37b22c3cbe7"
May 26 11:55:43.371843 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:55:43.371823    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"114d5cdc6116bd63a47e3ecee497cc7c9db5c2cd5f6075b4eaa8f37b22c3cbe7\": container with ID starting with 114d5cdc6116bd63a47e3ecee497cc7c9db5c2cd5f6075b4eaa8f37b22c3cbe7 not found: ID does not exist" containerID="114d5cdc6116bd63a47e3ecee497cc7c9db5c2cd5f6075b4eaa8f37b22c3cbe7"
May 26 11:55:43.371949 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:55:43.371851    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"114d5cdc6116bd63a47e3ecee497cc7c9db5c2cd5f6075b4eaa8f37b22c3cbe7"} err="failed to get container status \"114d5cdc6116bd63a47e3ecee497cc7c9db5c2cd5f6075b4eaa8f37b22c3cbe7\": rpc error: code = NotFound desc = could not find container \"114d5cdc6116bd63a47e3ecee497cc7c9db5c2cd5f6075b4eaa8f37b22c3cbe7\": container with ID starting with 114d5cdc6116bd63a47e3ecee497cc7c9db5c2cd5f6075b4eaa8f37b22c3cbe7 not found: ID does not exist"
May 26 11:55:43.371949 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:55:43.371872    2572 scope.go:117] "RemoveContainer" containerID="c5f89953ccd1141b95cd453fb99f697d69b23d9a6214a0955f57285aaa6536bb"
May 26 11:55:43.372213 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:55:43.372153    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5f89953ccd1141b95cd453fb99f697d69b23d9a6214a0955f57285aaa6536bb\": container with ID starting with c5f89953ccd1141b95cd453fb99f697d69b23d9a6214a0955f57285aaa6536bb not found: ID does not exist" containerID="c5f89953ccd1141b95cd453fb99f697d69b23d9a6214a0955f57285aaa6536bb"
May 26 11:55:43.372335 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:55:43.372206    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5f89953ccd1141b95cd453fb99f697d69b23d9a6214a0955f57285aaa6536bb"} err="failed to get container status \"c5f89953ccd1141b95cd453fb99f697d69b23d9a6214a0955f57285aaa6536bb\": rpc error: code = NotFound desc = could not find container \"c5f89953ccd1141b95cd453fb99f697d69b23d9a6214a0955f57285aaa6536bb\": container with ID starting with c5f89953ccd1141b95cd453fb99f697d69b23d9a6214a0955f57285aaa6536bb not found: ID does not exist"
May 26 11:55:43.372335 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:55:43.372230    2572 scope.go:117] "RemoveContainer" containerID="cea8c33e26f344652cff649ea777fcaa5050980811c9918470e2962549a5a5c5"
May 26 11:55:43.372489 ip-10-0-136-194 kubenswrapper[2572]: E0526 11:55:43.372469    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cea8c33e26f344652cff649ea777fcaa5050980811c9918470e2962549a5a5c5\": container with ID starting with cea8c33e26f344652cff649ea777fcaa5050980811c9918470e2962549a5a5c5 not found: ID does not exist" containerID="cea8c33e26f344652cff649ea777fcaa5050980811c9918470e2962549a5a5c5"
May 26 11:55:43.372556 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:55:43.372494    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cea8c33e26f344652cff649ea777fcaa5050980811c9918470e2962549a5a5c5"} err="failed to get container status \"cea8c33e26f344652cff649ea777fcaa5050980811c9918470e2962549a5a5c5\": rpc error: code = NotFound desc = could not find container \"cea8c33e26f344652cff649ea777fcaa5050980811c9918470e2962549a5a5c5\": container with ID starting with cea8c33e26f344652cff649ea777fcaa5050980811c9918470e2962549a5a5c5 not found: ID does not exist"
May 26 11:55:43.372556 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:55:43.372511    2572 scope.go:117] "RemoveContainer" containerID="114d5cdc6116bd63a47e3ecee497cc7c9db5c2cd5f6075b4eaa8f37b22c3cbe7"
May 26 11:55:43.372815 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:55:43.372792    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"114d5cdc6116bd63a47e3ecee497cc7c9db5c2cd5f6075b4eaa8f37b22c3cbe7"} err="failed to get container status \"114d5cdc6116bd63a47e3ecee497cc7c9db5c2cd5f6075b4eaa8f37b22c3cbe7\": rpc error: code = NotFound desc = could not find container \"114d5cdc6116bd63a47e3ecee497cc7c9db5c2cd5f6075b4eaa8f37b22c3cbe7\": container with ID starting with 114d5cdc6116bd63a47e3ecee497cc7c9db5c2cd5f6075b4eaa8f37b22c3cbe7 not found: ID does not exist"
May 26 11:55:43.372815 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:55:43.372813    2572 scope.go:117] "RemoveContainer" containerID="c5f89953ccd1141b95cd453fb99f697d69b23d9a6214a0955f57285aaa6536bb"
May 26 11:55:43.373088 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:55:43.373067    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5f89953ccd1141b95cd453fb99f697d69b23d9a6214a0955f57285aaa6536bb"} err="failed to get container status \"c5f89953ccd1141b95cd453fb99f697d69b23d9a6214a0955f57285aaa6536bb\": rpc error: code = NotFound desc = could not find container \"c5f89953ccd1141b95cd453fb99f697d69b23d9a6214a0955f57285aaa6536bb\": container with ID starting with c5f89953ccd1141b95cd453fb99f697d69b23d9a6214a0955f57285aaa6536bb not found: ID does not exist"
May 26 11:55:43.373088 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:55:43.373088    2572 scope.go:117] "RemoveContainer" containerID="cea8c33e26f344652cff649ea777fcaa5050980811c9918470e2962549a5a5c5"
May 26 11:55:43.373362 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:55:43.373340    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cea8c33e26f344652cff649ea777fcaa5050980811c9918470e2962549a5a5c5"} err="failed to get container status \"cea8c33e26f344652cff649ea777fcaa5050980811c9918470e2962549a5a5c5\": rpc error: code = NotFound desc = could not find container \"cea8c33e26f344652cff649ea777fcaa5050980811c9918470e2962549a5a5c5\": container with ID starting with cea8c33e26f344652cff649ea777fcaa5050980811c9918470e2962549a5a5c5 not found: ID does not exist"
May 26 11:55:43.414006 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:55:43.413977    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/61e6b55c-5d96-4566-8f1a-c35a4097c57f-dshm\") pod \"61e6b55c-5d96-4566-8f1a-c35a4097c57f\" (UID: \"61e6b55c-5d96-4566-8f1a-c35a4097c57f\") "
May 26 11:55:43.414129 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:55:43.414035    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/25b0394c-f00b-4f2e-acb4-853d2a617f79-home\") pod \"25b0394c-f00b-4f2e-acb4-853d2a617f79\" (UID: \"25b0394c-f00b-4f2e-acb4-853d2a617f79\") "
May 26 11:55:43.414129 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:55:43.414080    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61e6b55c-5d96-4566-8f1a-c35a4097c57f-model-cache\") pod \"61e6b55c-5d96-4566-8f1a-c35a4097c57f\" (UID: \"61e6b55c-5d96-4566-8f1a-c35a4097c57f\") "
May 26 11:55:43.414129 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:55:43.414121    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/25b0394c-f00b-4f2e-acb4-853d2a617f79-kserve-provision-location\") pod \"25b0394c-f00b-4f2e-acb4-853d2a617f79\" (UID: \"25b0394c-f00b-4f2e-acb4-853d2a617f79\") "
May 26 11:55:43.414333 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:55:43.414162    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d9qhj\" (UniqueName: \"kubernetes.io/projected/25b0394c-f00b-4f2e-acb4-853d2a617f79-kube-api-access-d9qhj\") pod \"25b0394c-f00b-4f2e-acb4-853d2a617f79\" (UID: \"25b0394c-f00b-4f2e-acb4-853d2a617f79\") "
May 26 11:55:43.414333 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:55:43.414241    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/61e6b55c-5d96-4566-8f1a-c35a4097c57f-home\") pod \"61e6b55c-5d96-4566-8f1a-c35a4097c57f\" (UID: \"61e6b55c-5d96-4566-8f1a-c35a4097c57f\") "
May 26 11:55:43.414333 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:55:43.414271    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/25b0394c-f00b-4f2e-acb4-853d2a617f79-tmp-dir\") pod \"25b0394c-f00b-4f2e-acb4-853d2a617f79\" (UID: \"25b0394c-f00b-4f2e-acb4-853d2a617f79\") "
May 26 11:55:43.414333 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:55:43.414320    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lf4rh\" (UniqueName: \"kubernetes.io/projected/61e6b55c-5d96-4566-8f1a-c35a4097c57f-kube-api-access-lf4rh\") pod \"61e6b55c-5d96-4566-8f1a-c35a4097c57f\" (UID: \"61e6b55c-5d96-4566-8f1a-c35a4097c57f\") "
May 26 11:55:43.414556 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:55:43.414362    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/25b0394c-f00b-4f2e-acb4-853d2a617f79-tls-certs\") pod \"25b0394c-f00b-4f2e-acb4-853d2a617f79\" (UID: \"25b0394c-f00b-4f2e-acb4-853d2a617f79\") "
May 26 11:55:43.414556 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:55:43.414370    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61e6b55c-5d96-4566-8f1a-c35a4097c57f-model-cache" (OuterVolumeSpecName: "model-cache") pod "61e6b55c-5d96-4566-8f1a-c35a4097c57f" (UID: "61e6b55c-5d96-4566-8f1a-c35a4097c57f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 11:55:43.414556 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:55:43.414398    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/25b0394c-f00b-4f2e-acb4-853d2a617f79-model-cache\") pod \"25b0394c-f00b-4f2e-acb4-853d2a617f79\" (UID: \"25b0394c-f00b-4f2e-acb4-853d2a617f79\") "
May 26 11:55:43.414717 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:55:43.414638    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25b0394c-f00b-4f2e-acb4-853d2a617f79-model-cache" (OuterVolumeSpecName: "model-cache") pod "25b0394c-f00b-4f2e-acb4-853d2a617f79" (UID: "25b0394c-f00b-4f2e-acb4-853d2a617f79"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 11:55:43.414866 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:55:43.414840    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25b0394c-f00b-4f2e-acb4-853d2a617f79-home" (OuterVolumeSpecName: "home") pod "25b0394c-f00b-4f2e-acb4-853d2a617f79" (UID: "25b0394c-f00b-4f2e-acb4-853d2a617f79"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 11:55:43.414933 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:55:43.414907    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/25b0394c-f00b-4f2e-acb4-853d2a617f79-dshm\") pod \"25b0394c-f00b-4f2e-acb4-853d2a617f79\" (UID: \"25b0394c-f00b-4f2e-acb4-853d2a617f79\") "
May 26 11:55:43.414999 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:55:43.414949    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61e6b55c-5d96-4566-8f1a-c35a4097c57f-tls-certs\") pod \"61e6b55c-5d96-4566-8f1a-c35a4097c57f\" (UID: \"61e6b55c-5d96-4566-8f1a-c35a4097c57f\") "
May 26 11:55:43.414999 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:55:43.414978    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/61e6b55c-5d96-4566-8f1a-c35a4097c57f-kserve-provision-location\") pod \"61e6b55c-5d96-4566-8f1a-c35a4097c57f\" (UID: \"61e6b55c-5d96-4566-8f1a-c35a4097c57f\") "
May 26 11:55:43.415104 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:55:43.415023    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61e6b55c-5d96-4566-8f1a-c35a4097c57f-tmp-dir\") pod \"61e6b55c-5d96-4566-8f1a-c35a4097c57f\" (UID: \"61e6b55c-5d96-4566-8f1a-c35a4097c57f\") "
May 26 11:55:43.415380 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:55:43.415363    2572 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/25b0394c-f00b-4f2e-acb4-853d2a617f79-home\") on node \"ip-10-0-136-194.ec2.internal\" DevicePath \"\""
May 26 11:55:43.415475 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:55:43.415384    2572 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61e6b55c-5d96-4566-8f1a-c35a4097c57f-model-cache\") on node \"ip-10-0-136-194.ec2.internal\" DevicePath \"\""
May 26 11:55:43.415475 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:55:43.415402    2572 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/25b0394c-f00b-4f2e-acb4-853d2a617f79-model-cache\") on node \"ip-10-0-136-194.ec2.internal\" DevicePath \"\""
May 26 11:55:43.416689 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:55:43.416662    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61e6b55c-5d96-4566-8f1a-c35a4097c57f-dshm" (OuterVolumeSpecName: "dshm") pod "61e6b55c-5d96-4566-8f1a-c35a4097c57f" (UID: "61e6b55c-5d96-4566-8f1a-c35a4097c57f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 11:55:43.416792 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:55:43.416769    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61e6b55c-5d96-4566-8f1a-c35a4097c57f-home" (OuterVolumeSpecName: "home") pod "61e6b55c-5d96-4566-8f1a-c35a4097c57f" (UID: "61e6b55c-5d96-4566-8f1a-c35a4097c57f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 11:55:43.417179 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:55:43.416972    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61e6b55c-5d96-4566-8f1a-c35a4097c57f-kube-api-access-lf4rh" (OuterVolumeSpecName: "kube-api-access-lf4rh") pod "61e6b55c-5d96-4566-8f1a-c35a4097c57f" (UID: "61e6b55c-5d96-4566-8f1a-c35a4097c57f"). InnerVolumeSpecName "kube-api-access-lf4rh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 11:55:43.417179 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:55:43.417067    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25b0394c-f00b-4f2e-acb4-853d2a617f79-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "25b0394c-f00b-4f2e-acb4-853d2a617f79" (UID: "25b0394c-f00b-4f2e-acb4-853d2a617f79"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 11:55:43.419364 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:55:43.419285    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61e6b55c-5d96-4566-8f1a-c35a4097c57f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "61e6b55c-5d96-4566-8f1a-c35a4097c57f" (UID: "61e6b55c-5d96-4566-8f1a-c35a4097c57f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 11:55:43.419519 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:55:43.419493    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25b0394c-f00b-4f2e-acb4-853d2a617f79-dshm" (OuterVolumeSpecName: "dshm") pod "25b0394c-f00b-4f2e-acb4-853d2a617f79" (UID: "25b0394c-f00b-4f2e-acb4-853d2a617f79"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 11:55:43.419627 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:55:43.419576    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25b0394c-f00b-4f2e-acb4-853d2a617f79-kube-api-access-d9qhj" (OuterVolumeSpecName: "kube-api-access-d9qhj") pod "25b0394c-f00b-4f2e-acb4-853d2a617f79" (UID: "25b0394c-f00b-4f2e-acb4-853d2a617f79"). InnerVolumeSpecName "kube-api-access-d9qhj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 11:55:43.430164 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:55:43.430138    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61e6b55c-5d96-4566-8f1a-c35a4097c57f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "61e6b55c-5d96-4566-8f1a-c35a4097c57f" (UID: "61e6b55c-5d96-4566-8f1a-c35a4097c57f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 11:55:43.434528 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:55:43.434494    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25b0394c-f00b-4f2e-acb4-853d2a617f79-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "25b0394c-f00b-4f2e-acb4-853d2a617f79" (UID: "25b0394c-f00b-4f2e-acb4-853d2a617f79"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 11:55:43.448935 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:55:43.448896    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25b0394c-f00b-4f2e-acb4-853d2a617f79-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "25b0394c-f00b-4f2e-acb4-853d2a617f79" (UID: "25b0394c-f00b-4f2e-acb4-853d2a617f79"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 11:55:43.465238 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:55:43.465211    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61e6b55c-5d96-4566-8f1a-c35a4097c57f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "61e6b55c-5d96-4566-8f1a-c35a4097c57f" (UID: "61e6b55c-5d96-4566-8f1a-c35a4097c57f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 11:55:43.516246 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:55:43.516219    2572 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/61e6b55c-5d96-4566-8f1a-c35a4097c57f-dshm\") on node \"ip-10-0-136-194.ec2.internal\" DevicePath \"\""
May 26 11:55:43.516246 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:55:43.516243    2572 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/25b0394c-f00b-4f2e-acb4-853d2a617f79-kserve-provision-location\") on node \"ip-10-0-136-194.ec2.internal\" DevicePath \"\""
May 26 11:55:43.516734 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:55:43.516255    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d9qhj\" (UniqueName: \"kubernetes.io/projected/25b0394c-f00b-4f2e-acb4-853d2a617f79-kube-api-access-d9qhj\") on node \"ip-10-0-136-194.ec2.internal\" DevicePath \"\""
May 26 11:55:43.516734 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:55:43.516264    2572 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/61e6b55c-5d96-4566-8f1a-c35a4097c57f-home\") on node \"ip-10-0-136-194.ec2.internal\" DevicePath \"\""
May 26 11:55:43.516734 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:55:43.516273    2572 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/25b0394c-f00b-4f2e-acb4-853d2a617f79-tmp-dir\") on node \"ip-10-0-136-194.ec2.internal\" DevicePath \"\""
May 26 11:55:43.516734 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:55:43.516282    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lf4rh\" (UniqueName: \"kubernetes.io/projected/61e6b55c-5d96-4566-8f1a-c35a4097c57f-kube-api-access-lf4rh\") on node \"ip-10-0-136-194.ec2.internal\" DevicePath \"\""
May 26 11:55:43.516734 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:55:43.516290    2572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/25b0394c-f00b-4f2e-acb4-853d2a617f79-tls-certs\") on node \"ip-10-0-136-194.ec2.internal\" DevicePath \"\""
May 26 11:55:43.516734 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:55:43.516297    2572 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/25b0394c-f00b-4f2e-acb4-853d2a617f79-dshm\") on node \"ip-10-0-136-194.ec2.internal\" DevicePath \"\""
May 26 11:55:43.516734 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:55:43.516305    2572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61e6b55c-5d96-4566-8f1a-c35a4097c57f-tls-certs\") on node \"ip-10-0-136-194.ec2.internal\" DevicePath \"\""
May 26 11:55:43.516734 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:55:43.516313    2572 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/61e6b55c-5d96-4566-8f1a-c35a4097c57f-kserve-provision-location\") on node \"ip-10-0-136-194.ec2.internal\" DevicePath \"\""
May 26 11:55:43.516734 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:55:43.516321    2572 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61e6b55c-5d96-4566-8f1a-c35a4097c57f-tmp-dir\") on node \"ip-10-0-136-194.ec2.internal\" DevicePath \"\""
May 26 11:55:44.323464 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:55:44.323375    2572 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7bc7c9f5cf8xqlh"
May 26 11:55:44.323464 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:55:44.323418    2572 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-55hg57h"
May 26 11:55:44.346359 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:55:44.346324    2572 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7bc7c9f5cf8xqlh"]
May 26 11:55:44.349057 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:55:44.349029    2572 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-7bc7c9f5cf8xqlh"]
May 26 11:55:44.359990 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:55:44.359968    2572 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-55hg57h"]
May 26 11:55:44.366243 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:55:44.366223    2572 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-55hg57h"]
May 26 11:55:45.676805 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:55:45.676773    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="25b0394c-f00b-4f2e-acb4-853d2a617f79" path="/var/lib/kubelet/pods/25b0394c-f00b-4f2e-acb4-853d2a617f79/volumes"
May 26 11:55:45.677271 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:55:45.677256    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="61e6b55c-5d96-4566-8f1a-c35a4097c57f" path="/var/lib/kubelet/pods/61e6b55c-5d96-4566-8f1a-c35a4097c57f/volumes"
May 26 11:55:52.500439 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:55:52.500398    2572 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7955f5555b-4jdzr" podUID="9345be29-4f25-41c5-8489-c02aa4b7cea6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8001/health\": dial tcp 10.132.0.47:8001: connect: connection refused"
May 26 11:56:02.500957 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:56:02.500908    2572 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7955f5555b-4jdzr" podUID="9345be29-4f25-41c5-8489-c02aa4b7cea6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8001/health\": dial tcp 10.132.0.47:8001: connect: connection refused"
May 26 11:56:12.500403 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:56:12.500358    2572 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7955f5555b-4jdzr" podUID="9345be29-4f25-41c5-8489-c02aa4b7cea6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8001/health\": dial tcp 10.132.0.47:8001: connect: connection refused"
May 26 11:56:22.500553 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:56:22.500501    2572 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7955f5555b-4jdzr" podUID="9345be29-4f25-41c5-8489-c02aa4b7cea6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8001/health\": dial tcp 10.132.0.47:8001: connect: connection refused"
May 26 11:56:32.500920 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:56:32.500876    2572 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7955f5555b-4jdzr" podUID="9345be29-4f25-41c5-8489-c02aa4b7cea6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8001/health\": dial tcp 10.132.0.47:8001: connect: connection refused"
May 26 11:56:42.500980 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:56:42.500924    2572 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7955f5555b-4jdzr" podUID="9345be29-4f25-41c5-8489-c02aa4b7cea6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8001/health\": dial tcp 10.132.0.47:8001: connect: connection refused"
May 26 11:56:52.500869 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:56:52.500822    2572 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7955f5555b-4jdzr" podUID="9345be29-4f25-41c5-8489-c02aa4b7cea6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8001/health\": dial tcp 10.132.0.47:8001: connect: connection refused"
May 26 11:57:02.509744 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:57:02.509707    2572 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7955f5555b-4jdzr"
May 26 11:57:02.521623 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:57:02.521597    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7955f5555b-4jdzr"
May 26 11:59:37.754153 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:59:37.754123    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-5bfgg_c6373342-dd07-4497-a4f3-978ea9613b6e/console-operator/2.log"
May 26 11:59:37.760868 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:59:37.760833    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-5bfgg_c6373342-dd07-4497-a4f3-978ea9613b6e/console-operator/2.log"
May 26 11:59:37.761052 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:59:37.761036    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-js8cc_7264c907-f536-4719-905a-2bbda9e3c336/ovn-acl-logging/0.log"
May 26 11:59:37.767060 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:59:37.767039    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-js8cc_7264c907-f536-4719-905a-2bbda9e3c336/ovn-acl-logging/0.log"
May 26 11:59:37.774448 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:59:37.774426    2572 scope.go:117] "RemoveContainer" containerID="3afff25ad5eaab36ab40dbf91e9903cc3eff98a1a6caa8d9e780cc4bd1c084cf"
May 26 11:59:37.784586 ip-10-0-136-194 kubenswrapper[2572]: I0526 11:59:37.784563    2572 scope.go:117] "RemoveContainer" containerID="9843c7be8ce8e6f33efe657149ed2d6402e4771bdb359ae7cf5dd92ade20fa43"
May 26 12:00:29.286837 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:00:29.286755    2572 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-84f494576-mhtjs"]
May 26 12:00:29.289309 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:00:29.287119    2572 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-84f494576-mhtjs" podUID="ec94c1ea-a125-4069-a82d-9d5b6c2712ed" containerName="storage-initializer" containerID="cri-o://03bbd6d099de04026f99a1ccccbfe36429f0df9b00c7f3eabc1b149b45e1ba56" gracePeriod=30
May 26 12:00:29.292641 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:00:29.292617    2572 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-858b6cd9j49rn"]
May 26 12:00:29.293091 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:00:29.293040    2572 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-858b6cd9j49rn" podUID="d2039ec6-c16e-495f-927a-97103f09da12" containerName="main" containerID="cri-o://86721cc462c35ec1864092fa3b76d19334298dff17dc41e001a610ca161ad6b0" gracePeriod=30
May 26 12:00:29.293248 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:00:29.293162    2572 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-858b6cd9j49rn" podUID="d2039ec6-c16e-495f-927a-97103f09da12" containerName="tokenizer" containerID="cri-o://c3afc764fe2c252960e9df246eb7da69c01deefeb8f4d916a217d08dc8fd5b3a" gracePeriod=30
May 26 12:00:30.374712 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:00:30.374676    2572 generic.go:358] "Generic (PLEG): container finished" podID="d2039ec6-c16e-495f-927a-97103f09da12" containerID="86721cc462c35ec1864092fa3b76d19334298dff17dc41e001a610ca161ad6b0" exitCode=0
May 26 12:00:30.375104 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:00:30.374755    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-858b6cd9j49rn" event={"ID":"d2039ec6-c16e-495f-927a-97103f09da12","Type":"ContainerDied","Data":"86721cc462c35ec1864092fa3b76d19334298dff17dc41e001a610ca161ad6b0"}
May 26 12:00:30.851615 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:00:30.851591    2572 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-858b6cd9j49rn"
May 26 12:00:30.911442 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:00:30.911412    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2039ec6-c16e-495f-927a-97103f09da12-tokenizer-cache\") pod \"d2039ec6-c16e-495f-927a-97103f09da12\" (UID: \"d2039ec6-c16e-495f-927a-97103f09da12\") "
May 26 12:00:30.911619 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:00:30.911480    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cw6hw\" (UniqueName: \"kubernetes.io/projected/d2039ec6-c16e-495f-927a-97103f09da12-kube-api-access-cw6hw\") pod \"d2039ec6-c16e-495f-927a-97103f09da12\" (UID: \"d2039ec6-c16e-495f-927a-97103f09da12\") "
May 26 12:00:30.911619 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:00:30.911523    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d2039ec6-c16e-495f-927a-97103f09da12-kserve-provision-location\") pod \"d2039ec6-c16e-495f-927a-97103f09da12\" (UID: \"d2039ec6-c16e-495f-927a-97103f09da12\") "
May 26 12:00:30.911619 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:00:30.911615    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/d2039ec6-c16e-495f-927a-97103f09da12-tokenizer-tmp\") pod \"d2039ec6-c16e-495f-927a-97103f09da12\" (UID: \"d2039ec6-c16e-495f-927a-97103f09da12\") "
May 26 12:00:30.911742 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:00:30.911647    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/d2039ec6-c16e-495f-927a-97103f09da12-tokenizer-uds\") pod \"d2039ec6-c16e-495f-927a-97103f09da12\" (UID: \"d2039ec6-c16e-495f-927a-97103f09da12\") "
May 26 12:00:30.911742 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:00:30.911677    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2039ec6-c16e-495f-927a-97103f09da12-tls-certs\") pod \"d2039ec6-c16e-495f-927a-97103f09da12\" (UID: \"d2039ec6-c16e-495f-927a-97103f09da12\") "
May 26 12:00:30.911846 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:00:30.911818    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2039ec6-c16e-495f-927a-97103f09da12-tokenizer-cache" (OuterVolumeSpecName: "tokenizer-cache") pod "d2039ec6-c16e-495f-927a-97103f09da12" (UID: "d2039ec6-c16e-495f-927a-97103f09da12"). InnerVolumeSpecName "tokenizer-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 12:00:30.912007 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:00:30.911981    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2039ec6-c16e-495f-927a-97103f09da12-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "d2039ec6-c16e-495f-927a-97103f09da12" (UID: "d2039ec6-c16e-495f-927a-97103f09da12"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 12:00:30.912064 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:00:30.912012    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2039ec6-c16e-495f-927a-97103f09da12-tokenizer-tmp" (OuterVolumeSpecName: "tokenizer-tmp") pod "d2039ec6-c16e-495f-927a-97103f09da12" (UID: "d2039ec6-c16e-495f-927a-97103f09da12"). InnerVolumeSpecName "tokenizer-tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 12:00:30.912115 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:00:30.912068    2572 reconciler_common.go:299] "Volume detached for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2039ec6-c16e-495f-927a-97103f09da12-tokenizer-cache\") on node \"ip-10-0-136-194.ec2.internal\" DevicePath \"\""
May 26 12:00:30.912303 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:00:30.912283    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2039ec6-c16e-495f-927a-97103f09da12-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d2039ec6-c16e-495f-927a-97103f09da12" (UID: "d2039ec6-c16e-495f-927a-97103f09da12"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 12:00:30.913711 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:00:30.913691    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2039ec6-c16e-495f-927a-97103f09da12-kube-api-access-cw6hw" (OuterVolumeSpecName: "kube-api-access-cw6hw") pod "d2039ec6-c16e-495f-927a-97103f09da12" (UID: "d2039ec6-c16e-495f-927a-97103f09da12"). InnerVolumeSpecName "kube-api-access-cw6hw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 12:00:30.913821 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:00:30.913770    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2039ec6-c16e-495f-927a-97103f09da12-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d2039ec6-c16e-495f-927a-97103f09da12" (UID: "d2039ec6-c16e-495f-927a-97103f09da12"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 12:00:31.012614 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:00:31.012578    2572 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/d2039ec6-c16e-495f-927a-97103f09da12-tokenizer-uds\") on node \"ip-10-0-136-194.ec2.internal\" DevicePath \"\""
May 26 12:00:31.012614 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:00:31.012607    2572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2039ec6-c16e-495f-927a-97103f09da12-tls-certs\") on node \"ip-10-0-136-194.ec2.internal\" DevicePath \"\""
May 26 12:00:31.012614 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:00:31.012617    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cw6hw\" (UniqueName: \"kubernetes.io/projected/d2039ec6-c16e-495f-927a-97103f09da12-kube-api-access-cw6hw\") on node \"ip-10-0-136-194.ec2.internal\" DevicePath \"\""
May 26 12:00:31.012872 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:00:31.012626    2572 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d2039ec6-c16e-495f-927a-97103f09da12-kserve-provision-location\") on node \"ip-10-0-136-194.ec2.internal\" DevicePath \"\""
May 26 12:00:31.012872 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:00:31.012636    2572 reconciler_common.go:299] "Volume detached for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/d2039ec6-c16e-495f-927a-97103f09da12-tokenizer-tmp\") on node \"ip-10-0-136-194.ec2.internal\" DevicePath \"\""
May 26 12:00:31.379657 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:00:31.379573    2572 generic.go:358] "Generic (PLEG): container finished" podID="d2039ec6-c16e-495f-927a-97103f09da12" containerID="c3afc764fe2c252960e9df246eb7da69c01deefeb8f4d916a217d08dc8fd5b3a" exitCode=0
May 26 12:00:31.379657 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:00:31.379633    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-858b6cd9j49rn" event={"ID":"d2039ec6-c16e-495f-927a-97103f09da12","Type":"ContainerDied","Data":"c3afc764fe2c252960e9df246eb7da69c01deefeb8f4d916a217d08dc8fd5b3a"}
May 26 12:00:31.380202 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:00:31.379655    2572 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-858b6cd9j49rn"
May 26 12:00:31.380202 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:00:31.379671    2572 scope.go:117] "RemoveContainer" containerID="c3afc764fe2c252960e9df246eb7da69c01deefeb8f4d916a217d08dc8fd5b3a"
May 26 12:00:31.380202 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:00:31.379659    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-858b6cd9j49rn" event={"ID":"d2039ec6-c16e-495f-927a-97103f09da12","Type":"ContainerDied","Data":"8f550c1ddd29128ec4a97805299bee6e444e8745ef838296da0c2283117ebe06"}
May 26 12:00:31.388896 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:00:31.388879    2572 scope.go:117] "RemoveContainer" containerID="86721cc462c35ec1864092fa3b76d19334298dff17dc41e001a610ca161ad6b0"
May 26 12:00:31.396353 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:00:31.396335    2572 scope.go:117] "RemoveContainer" containerID="caad9e3b463dc44220995a7fe867859fd8372fe695054f9b2c10f08a4d383004"
May 26 12:00:31.404212 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:00:31.404172    2572 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-858b6cd9j49rn"]
May 26 12:00:31.404626 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:00:31.404605    2572 scope.go:117] "RemoveContainer" containerID="c3afc764fe2c252960e9df246eb7da69c01deefeb8f4d916a217d08dc8fd5b3a"
May 26 12:00:31.404922 ip-10-0-136-194 kubenswrapper[2572]: E0526 12:00:31.404904    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3afc764fe2c252960e9df246eb7da69c01deefeb8f4d916a217d08dc8fd5b3a\": container with ID starting with c3afc764fe2c252960e9df246eb7da69c01deefeb8f4d916a217d08dc8fd5b3a not found: ID does not exist" containerID="c3afc764fe2c252960e9df246eb7da69c01deefeb8f4d916a217d08dc8fd5b3a"
May 26 12:00:31.404977 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:00:31.404935    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3afc764fe2c252960e9df246eb7da69c01deefeb8f4d916a217d08dc8fd5b3a"} err="failed to get container status \"c3afc764fe2c252960e9df246eb7da69c01deefeb8f4d916a217d08dc8fd5b3a\": rpc error: code = NotFound desc = could not find container \"c3afc764fe2c252960e9df246eb7da69c01deefeb8f4d916a217d08dc8fd5b3a\": container with ID starting with c3afc764fe2c252960e9df246eb7da69c01deefeb8f4d916a217d08dc8fd5b3a not found: ID does not exist"
May 26 12:00:31.404977 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:00:31.404955    2572 scope.go:117] "RemoveContainer" containerID="86721cc462c35ec1864092fa3b76d19334298dff17dc41e001a610ca161ad6b0"
May 26 12:00:31.405245 ip-10-0-136-194 kubenswrapper[2572]: E0526 12:00:31.405222    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86721cc462c35ec1864092fa3b76d19334298dff17dc41e001a610ca161ad6b0\": container with ID starting with 86721cc462c35ec1864092fa3b76d19334298dff17dc41e001a610ca161ad6b0 not found: ID does not exist" containerID="86721cc462c35ec1864092fa3b76d19334298dff17dc41e001a610ca161ad6b0"
May 26 12:00:31.405307 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:00:31.405252    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86721cc462c35ec1864092fa3b76d19334298dff17dc41e001a610ca161ad6b0"} err="failed to get container status \"86721cc462c35ec1864092fa3b76d19334298dff17dc41e001a610ca161ad6b0\": rpc error: code = NotFound desc = could not find container \"86721cc462c35ec1864092fa3b76d19334298dff17dc41e001a610ca161ad6b0\": container with ID starting with 86721cc462c35ec1864092fa3b76d19334298dff17dc41e001a610ca161ad6b0 not found: ID does not exist"
May 26 12:00:31.405307 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:00:31.405270    2572 scope.go:117] "RemoveContainer" containerID="caad9e3b463dc44220995a7fe867859fd8372fe695054f9b2c10f08a4d383004"
May 26 12:00:31.405496 ip-10-0-136-194 kubenswrapper[2572]: E0526 12:00:31.405481    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"caad9e3b463dc44220995a7fe867859fd8372fe695054f9b2c10f08a4d383004\": container with ID starting with caad9e3b463dc44220995a7fe867859fd8372fe695054f9b2c10f08a4d383004 not found: ID does not exist" containerID="caad9e3b463dc44220995a7fe867859fd8372fe695054f9b2c10f08a4d383004"
May 26 12:00:31.405562 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:00:31.405499    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"caad9e3b463dc44220995a7fe867859fd8372fe695054f9b2c10f08a4d383004"} err="failed to get container status \"caad9e3b463dc44220995a7fe867859fd8372fe695054f9b2c10f08a4d383004\": rpc error: code = NotFound desc = could not find container \"caad9e3b463dc44220995a7fe867859fd8372fe695054f9b2c10f08a4d383004\": container with ID starting with caad9e3b463dc44220995a7fe867859fd8372fe695054f9b2c10f08a4d383004 not found: ID does not exist"
May 26 12:00:31.409402 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:00:31.409375    2572 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-858b6cd9j49rn"]
May 26 12:00:31.676108 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:00:31.676074    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d2039ec6-c16e-495f-927a-97103f09da12" path="/var/lib/kubelet/pods/d2039ec6-c16e-495f-927a-97103f09da12/volumes"
May 26 12:00:39.433387 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:00:39.433356    2572 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-5474d9f6f4-67rv4"]
May 26 12:00:39.433777 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:00:39.433763    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25b0394c-f00b-4f2e-acb4-853d2a617f79" containerName="llm-d-routing-sidecar"
May 26 12:00:39.433831 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:00:39.433779    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="25b0394c-f00b-4f2e-acb4-853d2a617f79" containerName="llm-d-routing-sidecar"
May 26 12:00:39.433831 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:00:39.433798    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25b0394c-f00b-4f2e-acb4-853d2a617f79" containerName="main"
May 26 12:00:39.433831 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:00:39.433803    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="25b0394c-f00b-4f2e-acb4-853d2a617f79" containerName="main"
May 26 12:00:39.433831 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:00:39.433815    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2039ec6-c16e-495f-927a-97103f09da12" containerName="main"
May 26 12:00:39.433831 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:00:39.433824    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2039ec6-c16e-495f-927a-97103f09da12" containerName="main"
May 26 12:00:39.434007 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:00:39.433842    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2039ec6-c16e-495f-927a-97103f09da12" containerName="storage-initializer"
May 26 12:00:39.434007 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:00:39.433848    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2039ec6-c16e-495f-927a-97103f09da12" containerName="storage-initializer"
May 26 12:00:39.434007 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:00:39.433856    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2039ec6-c16e-495f-927a-97103f09da12" containerName="tokenizer"
May 26 12:00:39.434007 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:00:39.433861    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2039ec6-c16e-495f-927a-97103f09da12" containerName="tokenizer"
May 26 12:00:39.434007 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:00:39.433869    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25b0394c-f00b-4f2e-acb4-853d2a617f79" containerName="storage-initializer"
May 26 12:00:39.434007 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:00:39.433875    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="25b0394c-f00b-4f2e-acb4-853d2a617f79" containerName="storage-initializer"
May 26 12:00:39.434007 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:00:39.433881    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="61e6b55c-5d96-4566-8f1a-c35a4097c57f" containerName="main"
May 26 12:00:39.434007 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:00:39.433887    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="61e6b55c-5d96-4566-8f1a-c35a4097c57f" containerName="main"
May 26 12:00:39.434007 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:00:39.433897    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="61e6b55c-5d96-4566-8f1a-c35a4097c57f" containerName="storage-initializer"
May 26 12:00:39.434007 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:00:39.433905    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="61e6b55c-5d96-4566-8f1a-c35a4097c57f" containerName="storage-initializer"
May 26 12:00:39.434007 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:00:39.433996    2572 memory_manager.go:356] "RemoveStaleState removing state" podUID="25b0394c-f00b-4f2e-acb4-853d2a617f79" containerName="main"
May 26 12:00:39.434007 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:00:39.434007    2572 memory_manager.go:356] "RemoveStaleState removing state" podUID="25b0394c-f00b-4f2e-acb4-853d2a617f79" containerName="llm-d-routing-sidecar"
May 26 12:00:39.434007 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:00:39.434014    2572 memory_manager.go:356] "RemoveStaleState removing state" podUID="61e6b55c-5d96-4566-8f1a-c35a4097c57f" containerName="main"
May 26 12:00:39.434450 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:00:39.434022    2572 memory_manager.go:356] "RemoveStaleState removing state" podUID="d2039ec6-c16e-495f-927a-97103f09da12" containerName="tokenizer"
May 26 12:00:39.434450 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:00:39.434028    2572 memory_manager.go:356] "RemoveStaleState removing state" podUID="d2039ec6-c16e-495f-927a-97103f09da12" containerName="main"
May 26 12:00:39.438152 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:00:39.438137    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5474d9f6f4-67rv4"
May 26 12:00:39.440835 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:00:39.440815    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"stop-feature-test-kserve-self-signed-certs\""
May 26 12:00:39.452396 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:00:39.452375    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-5474d9f6f4-67rv4"]
May 26 12:00:39.496384 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:00:39.496355    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0df0deec-ab10-4a22-95d5-e9d972bf5ea3-dshm\") pod \"stop-feature-test-kserve-5474d9f6f4-67rv4\" (UID: \"0df0deec-ab10-4a22-95d5-e9d972bf5ea3\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5474d9f6f4-67rv4"
May 26 12:00:39.496384 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:00:39.496385    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0df0deec-ab10-4a22-95d5-e9d972bf5ea3-tls-certs\") pod \"stop-feature-test-kserve-5474d9f6f4-67rv4\" (UID: \"0df0deec-ab10-4a22-95d5-e9d972bf5ea3\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5474d9f6f4-67rv4"
May 26 12:00:39.496580 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:00:39.496418    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjbhg\" (UniqueName: \"kubernetes.io/projected/0df0deec-ab10-4a22-95d5-e9d972bf5ea3-kube-api-access-fjbhg\") pod \"stop-feature-test-kserve-5474d9f6f4-67rv4\" (UID: \"0df0deec-ab10-4a22-95d5-e9d972bf5ea3\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5474d9f6f4-67rv4"
May 26 12:00:39.496580 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:00:39.496518    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0df0deec-ab10-4a22-95d5-e9d972bf5ea3-kserve-provision-location\") pod \"stop-feature-test-kserve-5474d9f6f4-67rv4\" (UID: \"0df0deec-ab10-4a22-95d5-e9d972bf5ea3\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5474d9f6f4-67rv4"
May 26 12:00:39.496580 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:00:39.496557    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0df0deec-ab10-4a22-95d5-e9d972bf5ea3-model-cache\") pod \"stop-feature-test-kserve-5474d9f6f4-67rv4\" (UID: \"0df0deec-ab10-4a22-95d5-e9d972bf5ea3\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5474d9f6f4-67rv4"
May 26 12:00:39.496673 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:00:39.496598    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0df0deec-ab10-4a22-95d5-e9d972bf5ea3-tmp-dir\") pod \"stop-feature-test-kserve-5474d9f6f4-67rv4\" (UID: \"0df0deec-ab10-4a22-95d5-e9d972bf5ea3\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5474d9f6f4-67rv4"
May 26 12:00:39.496673 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:00:39.496667    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0df0deec-ab10-4a22-95d5-e9d972bf5ea3-home\") pod \"stop-feature-test-kserve-5474d9f6f4-67rv4\" (UID: \"0df0deec-ab10-4a22-95d5-e9d972bf5ea3\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5474d9f6f4-67rv4"
May 26 12:00:39.597518 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:00:39.597472    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0df0deec-ab10-4a22-95d5-e9d972bf5ea3-kserve-provision-location\") pod \"stop-feature-test-kserve-5474d9f6f4-67rv4\" (UID: \"0df0deec-ab10-4a22-95d5-e9d972bf5ea3\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5474d9f6f4-67rv4"
May 26 12:00:39.597717 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:00:39.597533    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0df0deec-ab10-4a22-95d5-e9d972bf5ea3-model-cache\") pod \"stop-feature-test-kserve-5474d9f6f4-67rv4\" (UID: \"0df0deec-ab10-4a22-95d5-e9d972bf5ea3\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5474d9f6f4-67rv4"
May 26 12:00:39.597717 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:00:39.597568    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0df0deec-ab10-4a22-95d5-e9d972bf5ea3-tmp-dir\") pod \"stop-feature-test-kserve-5474d9f6f4-67rv4\" (UID: \"0df0deec-ab10-4a22-95d5-e9d972bf5ea3\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5474d9f6f4-67rv4"
May 26 12:00:39.597717 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:00:39.597639    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0df0deec-ab10-4a22-95d5-e9d972bf5ea3-home\") pod \"stop-feature-test-kserve-5474d9f6f4-67rv4\" (UID: \"0df0deec-ab10-4a22-95d5-e9d972bf5ea3\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5474d9f6f4-67rv4"
May 26 12:00:39.597899 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:00:39.597718    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0df0deec-ab10-4a22-95d5-e9d972bf5ea3-dshm\") pod \"stop-feature-test-kserve-5474d9f6f4-67rv4\" (UID: \"0df0deec-ab10-4a22-95d5-e9d972bf5ea3\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5474d9f6f4-67rv4"
May 26 12:00:39.597899 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:00:39.597743    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0df0deec-ab10-4a22-95d5-e9d972bf5ea3-tls-certs\") pod \"stop-feature-test-kserve-5474d9f6f4-67rv4\" (UID: \"0df0deec-ab10-4a22-95d5-e9d972bf5ea3\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5474d9f6f4-67rv4"
May 26 12:00:39.597899 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:00:39.597785    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-fjbhg\" (UniqueName: \"kubernetes.io/projected/0df0deec-ab10-4a22-95d5-e9d972bf5ea3-kube-api-access-fjbhg\") pod \"stop-feature-test-kserve-5474d9f6f4-67rv4\" (UID: \"0df0deec-ab10-4a22-95d5-e9d972bf5ea3\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5474d9f6f4-67rv4"
May 26 12:00:39.598058 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:00:39.597898    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0df0deec-ab10-4a22-95d5-e9d972bf5ea3-model-cache\") pod \"stop-feature-test-kserve-5474d9f6f4-67rv4\" (UID: \"0df0deec-ab10-4a22-95d5-e9d972bf5ea3\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5474d9f6f4-67rv4"
May 26 12:00:39.598058 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:00:39.597949    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0df0deec-ab10-4a22-95d5-e9d972bf5ea3-tmp-dir\") pod \"stop-feature-test-kserve-5474d9f6f4-67rv4\" (UID: \"0df0deec-ab10-4a22-95d5-e9d972bf5ea3\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5474d9f6f4-67rv4"
May 26 12:00:39.598058 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:00:39.597984    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0df0deec-ab10-4a22-95d5-e9d972bf5ea3-kserve-provision-location\") pod \"stop-feature-test-kserve-5474d9f6f4-67rv4\" (UID: \"0df0deec-ab10-4a22-95d5-e9d972bf5ea3\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5474d9f6f4-67rv4"
May 26 12:00:39.598243 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:00:39.598072    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0df0deec-ab10-4a22-95d5-e9d972bf5ea3-home\") pod \"stop-feature-test-kserve-5474d9f6f4-67rv4\" (UID: \"0df0deec-ab10-4a22-95d5-e9d972bf5ea3\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5474d9f6f4-67rv4"
May 26 12:00:39.600074 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:00:39.600046    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0df0deec-ab10-4a22-95d5-e9d972bf5ea3-dshm\") pod \"stop-feature-test-kserve-5474d9f6f4-67rv4\" (UID: \"0df0deec-ab10-4a22-95d5-e9d972bf5ea3\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5474d9f6f4-67rv4"
May 26 12:00:39.600346 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:00:39.600328    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0df0deec-ab10-4a22-95d5-e9d972bf5ea3-tls-certs\") pod \"stop-feature-test-kserve-5474d9f6f4-67rv4\" (UID: \"0df0deec-ab10-4a22-95d5-e9d972bf5ea3\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5474d9f6f4-67rv4"
May 26 12:00:39.606916 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:00:39.606892    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjbhg\" (UniqueName: \"kubernetes.io/projected/0df0deec-ab10-4a22-95d5-e9d972bf5ea3-kube-api-access-fjbhg\") pod \"stop-feature-test-kserve-5474d9f6f4-67rv4\" (UID: \"0df0deec-ab10-4a22-95d5-e9d972bf5ea3\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5474d9f6f4-67rv4"
May 26 12:00:39.749062 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:00:39.748966    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5474d9f6f4-67rv4"
May 26 12:00:39.884317 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:00:39.884119    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-5474d9f6f4-67rv4"]
May 26 12:00:39.886637 ip-10-0-136-194 kubenswrapper[2572]: W0526 12:00:39.886609    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0df0deec_ab10_4a22_95d5_e9d972bf5ea3.slice/crio-422efdd7fb80c2c23608066d363547b972dba857e0cf59ba88a685b6883b0934 WatchSource:0}: Error finding container 422efdd7fb80c2c23608066d363547b972dba857e0cf59ba88a685b6883b0934: Status 404 returned error can't find the container with id 422efdd7fb80c2c23608066d363547b972dba857e0cf59ba88a685b6883b0934
May 26 12:00:39.888836 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:00:39.888816    2572 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 26 12:00:40.415596 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:00:40.415560    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5474d9f6f4-67rv4" event={"ID":"0df0deec-ab10-4a22-95d5-e9d972bf5ea3","Type":"ContainerStarted","Data":"dd77c6dbbeb1136e4718d0d465d66129348012bf630dae933ce56ea9b19a0e82"}
May 26 12:00:40.415773 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:00:40.415605    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5474d9f6f4-67rv4" event={"ID":"0df0deec-ab10-4a22-95d5-e9d972bf5ea3","Type":"ContainerStarted","Data":"422efdd7fb80c2c23608066d363547b972dba857e0cf59ba88a685b6883b0934"}
May 26 12:00:59.467779 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:00:59.467752    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_precise-prefix-cache-test-kserve-84f494576-mhtjs_ec94c1ea-a125-4069-a82d-9d5b6c2712ed/storage-initializer/0.log"
May 26 12:00:59.468138 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:00:59.467821    2572 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-84f494576-mhtjs"
May 26 12:00:59.485125 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:00:59.485102    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_precise-prefix-cache-test-kserve-84f494576-mhtjs_ec94c1ea-a125-4069-a82d-9d5b6c2712ed/storage-initializer/0.log"
May 26 12:00:59.485303 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:00:59.485142    2572 generic.go:358] "Generic (PLEG): container finished" podID="ec94c1ea-a125-4069-a82d-9d5b6c2712ed" containerID="03bbd6d099de04026f99a1ccccbfe36429f0df9b00c7f3eabc1b149b45e1ba56" exitCode=137
May 26 12:00:59.485303 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:00:59.485255    2572 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-84f494576-mhtjs"
May 26 12:00:59.485423 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:00:59.485379    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-84f494576-mhtjs" event={"ID":"ec94c1ea-a125-4069-a82d-9d5b6c2712ed","Type":"ContainerDied","Data":"03bbd6d099de04026f99a1ccccbfe36429f0df9b00c7f3eabc1b149b45e1ba56"}
May 26 12:00:59.485477 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:00:59.485428    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-84f494576-mhtjs" event={"ID":"ec94c1ea-a125-4069-a82d-9d5b6c2712ed","Type":"ContainerDied","Data":"8a21836305165d99b3c1007cbd57e61358c23077179812149d2dc8c730316cd1"}
May 26 12:00:59.485477 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:00:59.485453    2572 scope.go:117] "RemoveContainer" containerID="03bbd6d099de04026f99a1ccccbfe36429f0df9b00c7f3eabc1b149b45e1ba56"
May 26 12:00:59.527791 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:00:59.527766    2572 scope.go:117] "RemoveContainer" containerID="03bbd6d099de04026f99a1ccccbfe36429f0df9b00c7f3eabc1b149b45e1ba56"
May 26 12:00:59.528122 ip-10-0-136-194 kubenswrapper[2572]: E0526 12:00:59.528100    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03bbd6d099de04026f99a1ccccbfe36429f0df9b00c7f3eabc1b149b45e1ba56\": container with ID starting with 03bbd6d099de04026f99a1ccccbfe36429f0df9b00c7f3eabc1b149b45e1ba56 not found: ID does not exist" containerID="03bbd6d099de04026f99a1ccccbfe36429f0df9b00c7f3eabc1b149b45e1ba56"
May 26 12:00:59.528203 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:00:59.528137    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03bbd6d099de04026f99a1ccccbfe36429f0df9b00c7f3eabc1b149b45e1ba56"} err="failed to get container status \"03bbd6d099de04026f99a1ccccbfe36429f0df9b00c7f3eabc1b149b45e1ba56\": rpc error: code = NotFound desc = could not find container \"03bbd6d099de04026f99a1ccccbfe36429f0df9b00c7f3eabc1b149b45e1ba56\": container with ID starting with 03bbd6d099de04026f99a1ccccbfe36429f0df9b00c7f3eabc1b149b45e1ba56 not found: ID does not exist"
May 26 12:00:59.587155 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:00:59.587055    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec94c1ea-a125-4069-a82d-9d5b6c2712ed-model-cache\") pod \"ec94c1ea-a125-4069-a82d-9d5b6c2712ed\" (UID: \"ec94c1ea-a125-4069-a82d-9d5b6c2712ed\") "
May 26 12:00:59.587155 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:00:59.587114    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-646px\" (UniqueName: \"kubernetes.io/projected/ec94c1ea-a125-4069-a82d-9d5b6c2712ed-kube-api-access-646px\") pod \"ec94c1ea-a125-4069-a82d-9d5b6c2712ed\" (UID: \"ec94c1ea-a125-4069-a82d-9d5b6c2712ed\") "
May 26 12:00:59.587415 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:00:59.587166    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec94c1ea-a125-4069-a82d-9d5b6c2712ed-tmp-dir\") pod \"ec94c1ea-a125-4069-a82d-9d5b6c2712ed\" (UID: \"ec94c1ea-a125-4069-a82d-9d5b6c2712ed\") "
May 26 12:00:59.587415 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:00:59.587285    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec94c1ea-a125-4069-a82d-9d5b6c2712ed-home\") pod \"ec94c1ea-a125-4069-a82d-9d5b6c2712ed\" (UID: \"ec94c1ea-a125-4069-a82d-9d5b6c2712ed\") "
May 26 12:00:59.587415 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:00:59.587322    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ec94c1ea-a125-4069-a82d-9d5b6c2712ed-kserve-provision-location\") pod \"ec94c1ea-a125-4069-a82d-9d5b6c2712ed\" (UID: \"ec94c1ea-a125-4069-a82d-9d5b6c2712ed\") "
May 26 12:00:59.587415 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:00:59.587345    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec94c1ea-a125-4069-a82d-9d5b6c2712ed-tls-certs\") pod \"ec94c1ea-a125-4069-a82d-9d5b6c2712ed\" (UID: \"ec94c1ea-a125-4069-a82d-9d5b6c2712ed\") "
May 26 12:00:59.587415 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:00:59.587374    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec94c1ea-a125-4069-a82d-9d5b6c2712ed-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ec94c1ea-a125-4069-a82d-9d5b6c2712ed" (UID: "ec94c1ea-a125-4069-a82d-9d5b6c2712ed"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 12:00:59.587415 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:00:59.587389    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec94c1ea-a125-4069-a82d-9d5b6c2712ed-dshm\") pod \"ec94c1ea-a125-4069-a82d-9d5b6c2712ed\" (UID: \"ec94c1ea-a125-4069-a82d-9d5b6c2712ed\") "
May 26 12:00:59.587415 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:00:59.587390    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec94c1ea-a125-4069-a82d-9d5b6c2712ed-model-cache" (OuterVolumeSpecName: "model-cache") pod "ec94c1ea-a125-4069-a82d-9d5b6c2712ed" (UID: "ec94c1ea-a125-4069-a82d-9d5b6c2712ed"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 12:00:59.587786 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:00:59.587561    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec94c1ea-a125-4069-a82d-9d5b6c2712ed-home" (OuterVolumeSpecName: "home") pod "ec94c1ea-a125-4069-a82d-9d5b6c2712ed" (UID: "ec94c1ea-a125-4069-a82d-9d5b6c2712ed"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 12:00:59.587848 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:00:59.587795    2572 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec94c1ea-a125-4069-a82d-9d5b6c2712ed-tmp-dir\") on node \"ip-10-0-136-194.ec2.internal\" DevicePath \"\""
May 26 12:00:59.587848 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:00:59.587813    2572 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec94c1ea-a125-4069-a82d-9d5b6c2712ed-home\") on node \"ip-10-0-136-194.ec2.internal\" DevicePath \"\""
May 26 12:00:59.587848 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:00:59.587827    2572 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec94c1ea-a125-4069-a82d-9d5b6c2712ed-model-cache\") on node \"ip-10-0-136-194.ec2.internal\" DevicePath \"\""
May 26 12:00:59.589394 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:00:59.589368    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec94c1ea-a125-4069-a82d-9d5b6c2712ed-kube-api-access-646px" (OuterVolumeSpecName: "kube-api-access-646px") pod "ec94c1ea-a125-4069-a82d-9d5b6c2712ed" (UID: "ec94c1ea-a125-4069-a82d-9d5b6c2712ed"). InnerVolumeSpecName "kube-api-access-646px". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 12:00:59.589511 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:00:59.589453    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec94c1ea-a125-4069-a82d-9d5b6c2712ed-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ec94c1ea-a125-4069-a82d-9d5b6c2712ed" (UID: "ec94c1ea-a125-4069-a82d-9d5b6c2712ed"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 12:00:59.589677 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:00:59.589650    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec94c1ea-a125-4069-a82d-9d5b6c2712ed-dshm" (OuterVolumeSpecName: "dshm") pod "ec94c1ea-a125-4069-a82d-9d5b6c2712ed" (UID: "ec94c1ea-a125-4069-a82d-9d5b6c2712ed"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 12:00:59.624000 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:00:59.623955    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec94c1ea-a125-4069-a82d-9d5b6c2712ed-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ec94c1ea-a125-4069-a82d-9d5b6c2712ed" (UID: "ec94c1ea-a125-4069-a82d-9d5b6c2712ed"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 12:00:59.688632 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:00:59.688597    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-646px\" (UniqueName: \"kubernetes.io/projected/ec94c1ea-a125-4069-a82d-9d5b6c2712ed-kube-api-access-646px\") on node \"ip-10-0-136-194.ec2.internal\" DevicePath \"\""
May 26 12:00:59.688632 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:00:59.688632    2572 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ec94c1ea-a125-4069-a82d-9d5b6c2712ed-kserve-provision-location\") on node \"ip-10-0-136-194.ec2.internal\" DevicePath \"\""
May 26 12:00:59.688857 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:00:59.688646    2572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec94c1ea-a125-4069-a82d-9d5b6c2712ed-tls-certs\") on node \"ip-10-0-136-194.ec2.internal\" DevicePath \"\""
May 26 12:00:59.688857 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:00:59.688673    2572 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec94c1ea-a125-4069-a82d-9d5b6c2712ed-dshm\") on node \"ip-10-0-136-194.ec2.internal\" DevicePath \"\""
May 26 12:00:59.821115 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:00:59.821060    2572 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-84f494576-mhtjs"]
May 26 12:00:59.823132 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:00:59.823103    2572 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-84f494576-mhtjs"]
May 26 12:01:01.675904 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:01:01.675866    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ec94c1ea-a125-4069-a82d-9d5b6c2712ed" path="/var/lib/kubelet/pods/ec94c1ea-a125-4069-a82d-9d5b6c2712ed/volumes"
May 26 12:03:00.919468 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:03:00.919433    2572 generic.go:358] "Generic (PLEG): container finished" podID="0df0deec-ab10-4a22-95d5-e9d972bf5ea3" containerID="dd77c6dbbeb1136e4718d0d465d66129348012bf630dae933ce56ea9b19a0e82" exitCode=0
May 26 12:03:00.919468 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:03:00.919472    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5474d9f6f4-67rv4" event={"ID":"0df0deec-ab10-4a22-95d5-e9d972bf5ea3","Type":"ContainerDied","Data":"dd77c6dbbeb1136e4718d0d465d66129348012bf630dae933ce56ea9b19a0e82"}
May 26 12:03:01.924961 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:03:01.924925    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5474d9f6f4-67rv4" event={"ID":"0df0deec-ab10-4a22-95d5-e9d972bf5ea3","Type":"ContainerStarted","Data":"62d61d4981f5bea64844efe872a3afee4aa31ae5bd82a1fa690d76a4146ccac2"}
May 26 12:03:01.947110 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:03:01.947052    2572 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5474d9f6f4-67rv4" podStartSLOduration=142.947034507 podStartE2EDuration="2m22.947034507s" podCreationTimestamp="2026-05-26 12:00:39 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 12:03:01.945499743 +0000 UTC m=+2304.847710072" watchObservedRunningTime="2026-05-26 12:03:01.947034507 +0000 UTC m=+2304.849244837"
May 26 12:03:09.749912 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:03:09.749863    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5474d9f6f4-67rv4"
May 26 12:03:09.749912 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:03:09.749912    2572 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5474d9f6f4-67rv4"
May 26 12:03:09.751436 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:03:09.751402    2572 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5474d9f6f4-67rv4" podUID="0df0deec-ab10-4a22-95d5-e9d972bf5ea3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/health\": dial tcp 10.132.0.49:8000: connect: connection refused"
May 26 12:03:19.749343 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:03:19.749297    2572 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5474d9f6f4-67rv4" podUID="0df0deec-ab10-4a22-95d5-e9d972bf5ea3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/health\": dial tcp 10.132.0.49:8000: connect: connection refused"
May 26 12:03:29.749891 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:03:29.749792    2572 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5474d9f6f4-67rv4" podUID="0df0deec-ab10-4a22-95d5-e9d972bf5ea3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/health\": dial tcp 10.132.0.49:8000: connect: connection refused"
May 26 12:03:39.750161 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:03:39.750116    2572 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5474d9f6f4-67rv4" podUID="0df0deec-ab10-4a22-95d5-e9d972bf5ea3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/health\": dial tcp 10.132.0.49:8000: connect: connection refused"
May 26 12:03:49.750265 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:03:49.750225    2572 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5474d9f6f4-67rv4" podUID="0df0deec-ab10-4a22-95d5-e9d972bf5ea3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/health\": dial tcp 10.132.0.49:8000: connect: connection refused"
May 26 12:03:59.749736 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:03:59.749697    2572 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5474d9f6f4-67rv4" podUID="0df0deec-ab10-4a22-95d5-e9d972bf5ea3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/health\": dial tcp 10.132.0.49:8000: connect: connection refused"
May 26 12:04:09.749798 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:04:09.749746    2572 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5474d9f6f4-67rv4" podUID="0df0deec-ab10-4a22-95d5-e9d972bf5ea3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/health\": dial tcp 10.132.0.49:8000: connect: connection refused"
May 26 12:04:19.750100 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:04:19.750051    2572 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5474d9f6f4-67rv4" podUID="0df0deec-ab10-4a22-95d5-e9d972bf5ea3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/health\": dial tcp 10.132.0.49:8000: connect: connection refused"
May 26 12:04:29.749920 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:04:29.749870    2572 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5474d9f6f4-67rv4" podUID="0df0deec-ab10-4a22-95d5-e9d972bf5ea3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/health\": dial tcp 10.132.0.49:8000: connect: connection refused"
May 26 12:04:37.783970 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:04:37.783938    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-5bfgg_c6373342-dd07-4497-a4f3-978ea9613b6e/console-operator/2.log"
May 26 12:04:37.790000 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:04:37.789973    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-js8cc_7264c907-f536-4719-905a-2bbda9e3c336/ovn-acl-logging/0.log"
May 26 12:04:37.790139 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:04:37.790108    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-5bfgg_c6373342-dd07-4497-a4f3-978ea9613b6e/console-operator/2.log"
May 26 12:04:37.796176 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:04:37.796156    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-js8cc_7264c907-f536-4719-905a-2bbda9e3c336/ovn-acl-logging/0.log"
May 26 12:04:39.758978 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:04:39.758952    2572 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5474d9f6f4-67rv4"
May 26 12:04:39.766838 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:04:39.766813    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5474d9f6f4-67rv4"
May 26 12:04:40.706329 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:04:40.706294    2572 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-5474d9f6f4-67rv4"]
May 26 12:04:41.296880 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:04:41.296828    2572 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5474d9f6f4-67rv4" podUID="0df0deec-ab10-4a22-95d5-e9d972bf5ea3" containerName="main" containerID="cri-o://62d61d4981f5bea64844efe872a3afee4aa31ae5bd82a1fa690d76a4146ccac2" gracePeriod=30
May 26 12:04:54.827082 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:04:54.827003    2572 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-5474d9f6f4-kp2g6"]
May 26 12:04:54.827649 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:04:54.827631    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec94c1ea-a125-4069-a82d-9d5b6c2712ed" containerName="storage-initializer"
May 26 12:04:54.827693 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:04:54.827652    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec94c1ea-a125-4069-a82d-9d5b6c2712ed" containerName="storage-initializer"
May 26 12:04:54.827790 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:04:54.827773    2572 memory_manager.go:356] "RemoveStaleState removing state" podUID="ec94c1ea-a125-4069-a82d-9d5b6c2712ed" containerName="storage-initializer"
May 26 12:04:54.831412 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:04:54.831392    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5474d9f6f4-kp2g6"
May 26 12:04:54.842996 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:04:54.842972    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-5474d9f6f4-kp2g6"]
May 26 12:04:55.030276 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:04:55.030231    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d359422-07c3-419e-aa18-e74f8c38600e-model-cache\") pod \"stop-feature-test-kserve-5474d9f6f4-kp2g6\" (UID: \"9d359422-07c3-419e-aa18-e74f8c38600e\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5474d9f6f4-kp2g6"
May 26 12:04:55.030472 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:04:55.030299    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9d359422-07c3-419e-aa18-e74f8c38600e-home\") pod \"stop-feature-test-kserve-5474d9f6f4-kp2g6\" (UID: \"9d359422-07c3-419e-aa18-e74f8c38600e\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5474d9f6f4-kp2g6"
May 26 12:04:55.030472 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:04:55.030377    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-552f5\" (UniqueName: \"kubernetes.io/projected/9d359422-07c3-419e-aa18-e74f8c38600e-kube-api-access-552f5\") pod \"stop-feature-test-kserve-5474d9f6f4-kp2g6\" (UID: \"9d359422-07c3-419e-aa18-e74f8c38600e\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5474d9f6f4-kp2g6"
May 26 12:04:55.030472 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:04:55.030438    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d359422-07c3-419e-aa18-e74f8c38600e-tls-certs\") pod \"stop-feature-test-kserve-5474d9f6f4-kp2g6\" (UID: \"9d359422-07c3-419e-aa18-e74f8c38600e\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5474d9f6f4-kp2g6"
May 26 12:04:55.030733 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:04:55.030511    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d359422-07c3-419e-aa18-e74f8c38600e-tmp-dir\") pod \"stop-feature-test-kserve-5474d9f6f4-kp2g6\" (UID: \"9d359422-07c3-419e-aa18-e74f8c38600e\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5474d9f6f4-kp2g6"
May 26 12:04:55.030733 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:04:55.030535    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9d359422-07c3-419e-aa18-e74f8c38600e-kserve-provision-location\") pod \"stop-feature-test-kserve-5474d9f6f4-kp2g6\" (UID: \"9d359422-07c3-419e-aa18-e74f8c38600e\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5474d9f6f4-kp2g6"
May 26 12:04:55.030733 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:04:55.030569    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d359422-07c3-419e-aa18-e74f8c38600e-dshm\") pod \"stop-feature-test-kserve-5474d9f6f4-kp2g6\" (UID: \"9d359422-07c3-419e-aa18-e74f8c38600e\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5474d9f6f4-kp2g6"
May 26 12:04:55.131835 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:04:55.131746    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-552f5\" (UniqueName: \"kubernetes.io/projected/9d359422-07c3-419e-aa18-e74f8c38600e-kube-api-access-552f5\") pod \"stop-feature-test-kserve-5474d9f6f4-kp2g6\" (UID: \"9d359422-07c3-419e-aa18-e74f8c38600e\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5474d9f6f4-kp2g6"
May 26 12:04:55.131835 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:04:55.131811    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d359422-07c3-419e-aa18-e74f8c38600e-tls-certs\") pod \"stop-feature-test-kserve-5474d9f6f4-kp2g6\" (UID: \"9d359422-07c3-419e-aa18-e74f8c38600e\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5474d9f6f4-kp2g6"
May 26 12:04:55.132065 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:04:55.131850    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d359422-07c3-419e-aa18-e74f8c38600e-tmp-dir\") pod \"stop-feature-test-kserve-5474d9f6f4-kp2g6\" (UID: \"9d359422-07c3-419e-aa18-e74f8c38600e\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5474d9f6f4-kp2g6"
May 26 12:04:55.132065 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:04:55.131866    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9d359422-07c3-419e-aa18-e74f8c38600e-kserve-provision-location\") pod \"stop-feature-test-kserve-5474d9f6f4-kp2g6\" (UID: \"9d359422-07c3-419e-aa18-e74f8c38600e\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5474d9f6f4-kp2g6"
May 26 12:04:55.132065 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:04:55.131889    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d359422-07c3-419e-aa18-e74f8c38600e-dshm\") pod \"stop-feature-test-kserve-5474d9f6f4-kp2g6\" (UID: \"9d359422-07c3-419e-aa18-e74f8c38600e\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5474d9f6f4-kp2g6"
May 26 12:04:55.132065 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:04:55.132011    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d359422-07c3-419e-aa18-e74f8c38600e-model-cache\") pod \"stop-feature-test-kserve-5474d9f6f4-kp2g6\" (UID: \"9d359422-07c3-419e-aa18-e74f8c38600e\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5474d9f6f4-kp2g6"
May 26 12:04:55.132308 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:04:55.132081    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9d359422-07c3-419e-aa18-e74f8c38600e-home\") pod \"stop-feature-test-kserve-5474d9f6f4-kp2g6\" (UID: \"9d359422-07c3-419e-aa18-e74f8c38600e\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5474d9f6f4-kp2g6"
May 26 12:04:55.132553 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:04:55.132329    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d359422-07c3-419e-aa18-e74f8c38600e-tmp-dir\") pod \"stop-feature-test-kserve-5474d9f6f4-kp2g6\" (UID: \"9d359422-07c3-419e-aa18-e74f8c38600e\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5474d9f6f4-kp2g6"
May 26 12:04:55.132553 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:04:55.132382    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9d359422-07c3-419e-aa18-e74f8c38600e-kserve-provision-location\") pod \"stop-feature-test-kserve-5474d9f6f4-kp2g6\" (UID: \"9d359422-07c3-419e-aa18-e74f8c38600e\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5474d9f6f4-kp2g6"
May 26 12:04:55.132553 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:04:55.132438    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d359422-07c3-419e-aa18-e74f8c38600e-model-cache\") pod \"stop-feature-test-kserve-5474d9f6f4-kp2g6\" (UID: \"9d359422-07c3-419e-aa18-e74f8c38600e\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5474d9f6f4-kp2g6"
May 26 12:04:55.132553 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:04:55.132488    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9d359422-07c3-419e-aa18-e74f8c38600e-home\") pod \"stop-feature-test-kserve-5474d9f6f4-kp2g6\" (UID: \"9d359422-07c3-419e-aa18-e74f8c38600e\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5474d9f6f4-kp2g6"
May 26 12:04:55.134264 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:04:55.134237    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d359422-07c3-419e-aa18-e74f8c38600e-dshm\") pod \"stop-feature-test-kserve-5474d9f6f4-kp2g6\" (UID: \"9d359422-07c3-419e-aa18-e74f8c38600e\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5474d9f6f4-kp2g6"
May 26 12:04:55.134436 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:04:55.134418    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d359422-07c3-419e-aa18-e74f8c38600e-tls-certs\") pod \"stop-feature-test-kserve-5474d9f6f4-kp2g6\" (UID: \"9d359422-07c3-419e-aa18-e74f8c38600e\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5474d9f6f4-kp2g6"
May 26 12:04:55.140552 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:04:55.140534    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-552f5\" (UniqueName: \"kubernetes.io/projected/9d359422-07c3-419e-aa18-e74f8c38600e-kube-api-access-552f5\") pod \"stop-feature-test-kserve-5474d9f6f4-kp2g6\" (UID: \"9d359422-07c3-419e-aa18-e74f8c38600e\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5474d9f6f4-kp2g6"
May 26 12:04:55.142417 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:04:55.142398    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5474d9f6f4-kp2g6"
May 26 12:04:55.283929 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:04:55.283894    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-5474d9f6f4-kp2g6"]
May 26 12:04:55.293278 ip-10-0-136-194 kubenswrapper[2572]: W0526 12:04:55.293244    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d359422_07c3_419e_aa18_e74f8c38600e.slice/crio-53a446d5865a431514ee3a5d0213d7a47848a7fca3a7be77c03977f1111301be WatchSource:0}: Error finding container 53a446d5865a431514ee3a5d0213d7a47848a7fca3a7be77c03977f1111301be: Status 404 returned error can't find the container with id 53a446d5865a431514ee3a5d0213d7a47848a7fca3a7be77c03977f1111301be
May 26 12:04:55.346582 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:04:55.346554    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5474d9f6f4-kp2g6" event={"ID":"9d359422-07c3-419e-aa18-e74f8c38600e","Type":"ContainerStarted","Data":"53a446d5865a431514ee3a5d0213d7a47848a7fca3a7be77c03977f1111301be"}
May 26 12:04:56.353286 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:04:56.353244    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5474d9f6f4-kp2g6" event={"ID":"9d359422-07c3-419e-aa18-e74f8c38600e","Type":"ContainerStarted","Data":"0c0a4fe7ff814d7f376c002382a4c9fcb920a5b8f93678d68eb0d324da13f820"}
May 26 12:05:01.380473 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:05:01.380435    2572 generic.go:358] "Generic (PLEG): container finished" podID="9d359422-07c3-419e-aa18-e74f8c38600e" containerID="0c0a4fe7ff814d7f376c002382a4c9fcb920a5b8f93678d68eb0d324da13f820" exitCode=0
May 26 12:05:01.380884 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:05:01.380505    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5474d9f6f4-kp2g6" event={"ID":"9d359422-07c3-419e-aa18-e74f8c38600e","Type":"ContainerDied","Data":"0c0a4fe7ff814d7f376c002382a4c9fcb920a5b8f93678d68eb0d324da13f820"}
May 26 12:05:02.386092 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:05:02.386057    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5474d9f6f4-kp2g6" event={"ID":"9d359422-07c3-419e-aa18-e74f8c38600e","Type":"ContainerStarted","Data":"4277ec9534f8745a6c793021229d217d420d7fa63f495ee92c7578e5bcd61a3f"}
May 26 12:05:02.409276 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:05:02.409231    2572 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5474d9f6f4-kp2g6" podStartSLOduration=8.409215529 podStartE2EDuration="8.409215529s" podCreationTimestamp="2026-05-26 12:04:54 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 12:05:02.406412198 +0000 UTC m=+2425.308622528" watchObservedRunningTime="2026-05-26 12:05:02.409215529 +0000 UTC m=+2425.311425858"
May 26 12:05:05.142606 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:05:05.142572    2572 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5474d9f6f4-kp2g6"
May 26 12:05:05.142606 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:05:05.142612    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5474d9f6f4-kp2g6"
May 26 12:05:05.144154 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:05:05.144127    2572 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5474d9f6f4-kp2g6" podUID="9d359422-07c3-419e-aa18-e74f8c38600e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/health\": dial tcp 10.132.0.50:8000: connect: connection refused"
May 26 12:05:11.420917 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:05:11.420885    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_stop-feature-test-kserve-5474d9f6f4-67rv4_0df0deec-ab10-4a22-95d5-e9d972bf5ea3/main/0.log"
May 26 12:05:11.421382 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:05:11.421294    2572 generic.go:358] "Generic (PLEG): container finished" podID="0df0deec-ab10-4a22-95d5-e9d972bf5ea3" containerID="62d61d4981f5bea64844efe872a3afee4aa31ae5bd82a1fa690d76a4146ccac2" exitCode=137
May 26 12:05:11.421382 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:05:11.421353    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5474d9f6f4-67rv4" event={"ID":"0df0deec-ab10-4a22-95d5-e9d972bf5ea3","Type":"ContainerDied","Data":"62d61d4981f5bea64844efe872a3afee4aa31ae5bd82a1fa690d76a4146ccac2"}
May 26 12:05:11.542998 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:05:11.542968    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_stop-feature-test-kserve-5474d9f6f4-67rv4_0df0deec-ab10-4a22-95d5-e9d972bf5ea3/main/0.log"
May 26 12:05:11.543384 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:05:11.543366    2572 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5474d9f6f4-67rv4"
May 26 12:05:11.582245 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:05:11.582154    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0df0deec-ab10-4a22-95d5-e9d972bf5ea3-home\") pod \"0df0deec-ab10-4a22-95d5-e9d972bf5ea3\" (UID: \"0df0deec-ab10-4a22-95d5-e9d972bf5ea3\") "
May 26 12:05:11.582245 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:05:11.582219    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0df0deec-ab10-4a22-95d5-e9d972bf5ea3-tmp-dir\") pod \"0df0deec-ab10-4a22-95d5-e9d972bf5ea3\" (UID: \"0df0deec-ab10-4a22-95d5-e9d972bf5ea3\") "
May 26 12:05:11.582457 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:05:11.582249    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0df0deec-ab10-4a22-95d5-e9d972bf5ea3-model-cache\") pod \"0df0deec-ab10-4a22-95d5-e9d972bf5ea3\" (UID: \"0df0deec-ab10-4a22-95d5-e9d972bf5ea3\") "
May 26 12:05:11.582457 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:05:11.582290    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fjbhg\" (UniqueName: \"kubernetes.io/projected/0df0deec-ab10-4a22-95d5-e9d972bf5ea3-kube-api-access-fjbhg\") pod \"0df0deec-ab10-4a22-95d5-e9d972bf5ea3\" (UID: \"0df0deec-ab10-4a22-95d5-e9d972bf5ea3\") "
May 26 12:05:11.582457 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:05:11.582336    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0df0deec-ab10-4a22-95d5-e9d972bf5ea3-kserve-provision-location\") pod \"0df0deec-ab10-4a22-95d5-e9d972bf5ea3\" (UID: \"0df0deec-ab10-4a22-95d5-e9d972bf5ea3\") "
May 26 12:05:11.582457 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:05:11.582365    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0df0deec-ab10-4a22-95d5-e9d972bf5ea3-tls-certs\") pod \"0df0deec-ab10-4a22-95d5-e9d972bf5ea3\" (UID: \"0df0deec-ab10-4a22-95d5-e9d972bf5ea3\") "
May 26 12:05:11.582457 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:05:11.582395    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0df0deec-ab10-4a22-95d5-e9d972bf5ea3-dshm\") pod \"0df0deec-ab10-4a22-95d5-e9d972bf5ea3\" (UID: \"0df0deec-ab10-4a22-95d5-e9d972bf5ea3\") "
May 26 12:05:11.582785 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:05:11.582733    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0df0deec-ab10-4a22-95d5-e9d972bf5ea3-home" (OuterVolumeSpecName: "home") pod "0df0deec-ab10-4a22-95d5-e9d972bf5ea3" (UID: "0df0deec-ab10-4a22-95d5-e9d972bf5ea3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 12:05:11.586787 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:05:11.585098    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0df0deec-ab10-4a22-95d5-e9d972bf5ea3-model-cache" (OuterVolumeSpecName: "model-cache") pod "0df0deec-ab10-4a22-95d5-e9d972bf5ea3" (UID: "0df0deec-ab10-4a22-95d5-e9d972bf5ea3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 12:05:11.586787 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:05:11.585515    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0df0deec-ab10-4a22-95d5-e9d972bf5ea3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0df0deec-ab10-4a22-95d5-e9d972bf5ea3" (UID: "0df0deec-ab10-4a22-95d5-e9d972bf5ea3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 12:05:11.586787 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:05:11.585582    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0df0deec-ab10-4a22-95d5-e9d972bf5ea3-dshm" (OuterVolumeSpecName: "dshm") pod "0df0deec-ab10-4a22-95d5-e9d972bf5ea3" (UID: "0df0deec-ab10-4a22-95d5-e9d972bf5ea3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 12:05:11.587234 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:05:11.587208    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0df0deec-ab10-4a22-95d5-e9d972bf5ea3-kube-api-access-fjbhg" (OuterVolumeSpecName: "kube-api-access-fjbhg") pod "0df0deec-ab10-4a22-95d5-e9d972bf5ea3" (UID: "0df0deec-ab10-4a22-95d5-e9d972bf5ea3"). InnerVolumeSpecName "kube-api-access-fjbhg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 12:05:11.598072 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:05:11.598043    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0df0deec-ab10-4a22-95d5-e9d972bf5ea3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0df0deec-ab10-4a22-95d5-e9d972bf5ea3" (UID: "0df0deec-ab10-4a22-95d5-e9d972bf5ea3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 12:05:11.644104 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:05:11.644045    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0df0deec-ab10-4a22-95d5-e9d972bf5ea3-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0df0deec-ab10-4a22-95d5-e9d972bf5ea3" (UID: "0df0deec-ab10-4a22-95d5-e9d972bf5ea3"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 12:05:11.684064 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:05:11.684029    2572 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0df0deec-ab10-4a22-95d5-e9d972bf5ea3-home\") on node \"ip-10-0-136-194.ec2.internal\" DevicePath \"\""
May 26 12:05:11.684209 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:05:11.684067    2572 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0df0deec-ab10-4a22-95d5-e9d972bf5ea3-tmp-dir\") on node \"ip-10-0-136-194.ec2.internal\" DevicePath \"\""
May 26 12:05:11.684209 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:05:11.684085    2572 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0df0deec-ab10-4a22-95d5-e9d972bf5ea3-model-cache\") on node \"ip-10-0-136-194.ec2.internal\" DevicePath \"\""
May 26 12:05:11.684209 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:05:11.684100    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fjbhg\" (UniqueName: \"kubernetes.io/projected/0df0deec-ab10-4a22-95d5-e9d972bf5ea3-kube-api-access-fjbhg\") on node \"ip-10-0-136-194.ec2.internal\" DevicePath \"\""
May 26 12:05:11.684209 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:05:11.684115    2572 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0df0deec-ab10-4a22-95d5-e9d972bf5ea3-kserve-provision-location\") on node \"ip-10-0-136-194.ec2.internal\" DevicePath \"\""
May 26 12:05:11.684209 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:05:11.684130    2572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0df0deec-ab10-4a22-95d5-e9d972bf5ea3-tls-certs\") on node \"ip-10-0-136-194.ec2.internal\" DevicePath \"\""
May 26 12:05:11.684209 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:05:11.684144    2572 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0df0deec-ab10-4a22-95d5-e9d972bf5ea3-dshm\") on node \"ip-10-0-136-194.ec2.internal\" DevicePath \"\""
May 26 12:05:12.427310 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:05:12.427281    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_stop-feature-test-kserve-5474d9f6f4-67rv4_0df0deec-ab10-4a22-95d5-e9d972bf5ea3/main/0.log"
May 26 12:05:12.427769 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:05:12.427743    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5474d9f6f4-67rv4" event={"ID":"0df0deec-ab10-4a22-95d5-e9d972bf5ea3","Type":"ContainerDied","Data":"422efdd7fb80c2c23608066d363547b972dba857e0cf59ba88a685b6883b0934"}
May 26 12:05:12.427815 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:05:12.427797    2572 scope.go:117] "RemoveContainer" containerID="62d61d4981f5bea64844efe872a3afee4aa31ae5bd82a1fa690d76a4146ccac2"
May 26 12:05:12.427859 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:05:12.427751    2572 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5474d9f6f4-67rv4"
May 26 12:05:12.436702 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:05:12.436678    2572 scope.go:117] "RemoveContainer" containerID="dd77c6dbbeb1136e4718d0d465d66129348012bf630dae933ce56ea9b19a0e82"
May 26 12:05:12.450032 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:05:12.450005    2572 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-5474d9f6f4-67rv4"]
May 26 12:05:12.454872 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:05:12.454851    2572 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-5474d9f6f4-67rv4"]
May 26 12:05:13.676235 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:05:13.676201    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0df0deec-ab10-4a22-95d5-e9d972bf5ea3" path="/var/lib/kubelet/pods/0df0deec-ab10-4a22-95d5-e9d972bf5ea3/volumes"
May 26 12:05:15.143297 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:05:15.143261    2572 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5474d9f6f4-kp2g6" podUID="9d359422-07c3-419e-aa18-e74f8c38600e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/health\": dial tcp 10.132.0.50:8000: connect: connection refused"
May 26 12:05:25.143381 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:05:25.143329    2572 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5474d9f6f4-kp2g6" podUID="9d359422-07c3-419e-aa18-e74f8c38600e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/health\": dial tcp 10.132.0.50:8000: connect: connection refused"
May 26 12:05:35.143084 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:05:35.143037    2572 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5474d9f6f4-kp2g6" podUID="9d359422-07c3-419e-aa18-e74f8c38600e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/health\": dial tcp 10.132.0.50:8000: connect: connection refused"
May 26 12:05:45.143373 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:05:45.143321    2572 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5474d9f6f4-kp2g6" podUID="9d359422-07c3-419e-aa18-e74f8c38600e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/health\": dial tcp 10.132.0.50:8000: connect: connection refused"
May 26 12:05:55.143384 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:05:55.143339    2572 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5474d9f6f4-kp2g6" podUID="9d359422-07c3-419e-aa18-e74f8c38600e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/health\": dial tcp 10.132.0.50:8000: connect: connection refused"
May 26 12:06:05.143169 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:06:05.143128    2572 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5474d9f6f4-kp2g6" podUID="9d359422-07c3-419e-aa18-e74f8c38600e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/health\": dial tcp 10.132.0.50:8000: connect: connection refused"
May 26 12:06:15.143813 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:06:15.143758    2572 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5474d9f6f4-kp2g6" podUID="9d359422-07c3-419e-aa18-e74f8c38600e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/health\": dial tcp 10.132.0.50:8000: connect: connection refused"
May 26 12:06:25.143617 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:06:25.143515    2572 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5474d9f6f4-kp2g6" podUID="9d359422-07c3-419e-aa18-e74f8c38600e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/health\": dial tcp 10.132.0.50:8000: connect: connection refused"
May 26 12:06:35.152949 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:06:35.152918    2572 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5474d9f6f4-kp2g6"
May 26 12:06:35.168007 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:06:35.167974    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5474d9f6f4-kp2g6"
May 26 12:06:36.057294 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:06:36.057262    2572 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-5474d9f6f4-kp2g6"]
May 26 12:06:36.737004 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:06:36.736937    2572 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5474d9f6f4-kp2g6" podUID="9d359422-07c3-419e-aa18-e74f8c38600e" containerName="main" containerID="cri-o://4277ec9534f8745a6c793021229d217d420d7fa63f495ee92c7578e5bcd61a3f" gracePeriod=30
May 26 12:07:06.839548 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:07:06.839525    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_stop-feature-test-kserve-5474d9f6f4-kp2g6_9d359422-07c3-419e-aa18-e74f8c38600e/main/0.log"
May 26 12:07:06.839891 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:07:06.839874    2572 generic.go:358] "Generic (PLEG): container finished" podID="9d359422-07c3-419e-aa18-e74f8c38600e" containerID="4277ec9534f8745a6c793021229d217d420d7fa63f495ee92c7578e5bcd61a3f" exitCode=137
May 26 12:07:06.840032 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:07:06.839950    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5474d9f6f4-kp2g6" event={"ID":"9d359422-07c3-419e-aa18-e74f8c38600e","Type":"ContainerDied","Data":"4277ec9534f8745a6c793021229d217d420d7fa63f495ee92c7578e5bcd61a3f"}
May 26 12:07:06.974698 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:07:06.974680    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_stop-feature-test-kserve-5474d9f6f4-kp2g6_9d359422-07c3-419e-aa18-e74f8c38600e/main/0.log"
May 26 12:07:06.975074 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:07:06.975054    2572 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5474d9f6f4-kp2g6"
May 26 12:07:07.084312 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:07:07.084237    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d359422-07c3-419e-aa18-e74f8c38600e-dshm\") pod \"9d359422-07c3-419e-aa18-e74f8c38600e\" (UID: \"9d359422-07c3-419e-aa18-e74f8c38600e\") "
May 26 12:07:07.084312 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:07:07.084284    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d359422-07c3-419e-aa18-e74f8c38600e-tmp-dir\") pod \"9d359422-07c3-419e-aa18-e74f8c38600e\" (UID: \"9d359422-07c3-419e-aa18-e74f8c38600e\") "
May 26 12:07:07.084528 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:07:07.084328    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d359422-07c3-419e-aa18-e74f8c38600e-model-cache\") pod \"9d359422-07c3-419e-aa18-e74f8c38600e\" (UID: \"9d359422-07c3-419e-aa18-e74f8c38600e\") "
May 26 12:07:07.084528 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:07:07.084362    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d359422-07c3-419e-aa18-e74f8c38600e-tls-certs\") pod \"9d359422-07c3-419e-aa18-e74f8c38600e\" (UID: \"9d359422-07c3-419e-aa18-e74f8c38600e\") "
May 26 12:07:07.084528 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:07:07.084378    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9d359422-07c3-419e-aa18-e74f8c38600e-home\") pod \"9d359422-07c3-419e-aa18-e74f8c38600e\" (UID: \"9d359422-07c3-419e-aa18-e74f8c38600e\") "
May 26 12:07:07.084528 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:07:07.084398    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-552f5\" (UniqueName: \"kubernetes.io/projected/9d359422-07c3-419e-aa18-e74f8c38600e-kube-api-access-552f5\") pod \"9d359422-07c3-419e-aa18-e74f8c38600e\" (UID: \"9d359422-07c3-419e-aa18-e74f8c38600e\") "
May 26 12:07:07.084528 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:07:07.084416    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9d359422-07c3-419e-aa18-e74f8c38600e-kserve-provision-location\") pod \"9d359422-07c3-419e-aa18-e74f8c38600e\" (UID: \"9d359422-07c3-419e-aa18-e74f8c38600e\") "
May 26 12:07:07.084789 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:07:07.084648    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d359422-07c3-419e-aa18-e74f8c38600e-model-cache" (OuterVolumeSpecName: "model-cache") pod "9d359422-07c3-419e-aa18-e74f8c38600e" (UID: "9d359422-07c3-419e-aa18-e74f8c38600e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 12:07:07.084789 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:07:07.084730    2572 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d359422-07c3-419e-aa18-e74f8c38600e-model-cache\") on node \"ip-10-0-136-194.ec2.internal\" DevicePath \"\""
May 26 12:07:07.085082 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:07:07.085061    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d359422-07c3-419e-aa18-e74f8c38600e-home" (OuterVolumeSpecName: "home") pod "9d359422-07c3-419e-aa18-e74f8c38600e" (UID: "9d359422-07c3-419e-aa18-e74f8c38600e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 12:07:07.086472 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:07:07.086445    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d359422-07c3-419e-aa18-e74f8c38600e-dshm" (OuterVolumeSpecName: "dshm") pod "9d359422-07c3-419e-aa18-e74f8c38600e" (UID: "9d359422-07c3-419e-aa18-e74f8c38600e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 12:07:07.086867 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:07:07.086850    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d359422-07c3-419e-aa18-e74f8c38600e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9d359422-07c3-419e-aa18-e74f8c38600e" (UID: "9d359422-07c3-419e-aa18-e74f8c38600e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 12:07:07.087043 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:07:07.087027    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d359422-07c3-419e-aa18-e74f8c38600e-kube-api-access-552f5" (OuterVolumeSpecName: "kube-api-access-552f5") pod "9d359422-07c3-419e-aa18-e74f8c38600e" (UID: "9d359422-07c3-419e-aa18-e74f8c38600e"). InnerVolumeSpecName "kube-api-access-552f5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 12:07:07.097572 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:07:07.097548    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d359422-07c3-419e-aa18-e74f8c38600e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9d359422-07c3-419e-aa18-e74f8c38600e" (UID: "9d359422-07c3-419e-aa18-e74f8c38600e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 12:07:07.146530 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:07:07.146503    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d359422-07c3-419e-aa18-e74f8c38600e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "9d359422-07c3-419e-aa18-e74f8c38600e" (UID: "9d359422-07c3-419e-aa18-e74f8c38600e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 12:07:07.185796 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:07:07.185767    2572 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d359422-07c3-419e-aa18-e74f8c38600e-dshm\") on node \"ip-10-0-136-194.ec2.internal\" DevicePath \"\""
May 26 12:07:07.185902 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:07:07.185799    2572 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d359422-07c3-419e-aa18-e74f8c38600e-tmp-dir\") on node \"ip-10-0-136-194.ec2.internal\" DevicePath \"\""
May 26 12:07:07.185902 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:07:07.185815    2572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d359422-07c3-419e-aa18-e74f8c38600e-tls-certs\") on node \"ip-10-0-136-194.ec2.internal\" DevicePath \"\""
May 26 12:07:07.185902 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:07:07.185827    2572 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9d359422-07c3-419e-aa18-e74f8c38600e-home\") on node \"ip-10-0-136-194.ec2.internal\" DevicePath \"\""
May 26 12:07:07.185902 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:07:07.185842    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-552f5\" (UniqueName: \"kubernetes.io/projected/9d359422-07c3-419e-aa18-e74f8c38600e-kube-api-access-552f5\") on node \"ip-10-0-136-194.ec2.internal\" DevicePath \"\""
May 26 12:07:07.185902 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:07:07.185857    2572 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9d359422-07c3-419e-aa18-e74f8c38600e-kserve-provision-location\") on node \"ip-10-0-136-194.ec2.internal\" DevicePath \"\""
May 26 12:07:07.844937 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:07:07.844912    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_stop-feature-test-kserve-5474d9f6f4-kp2g6_9d359422-07c3-419e-aa18-e74f8c38600e/main/0.log"
May 26 12:07:07.845402 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:07:07.845307    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5474d9f6f4-kp2g6" event={"ID":"9d359422-07c3-419e-aa18-e74f8c38600e","Type":"ContainerDied","Data":"53a446d5865a431514ee3a5d0213d7a47848a7fca3a7be77c03977f1111301be"}
May 26 12:07:07.845402 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:07:07.845344    2572 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5474d9f6f4-kp2g6"
May 26 12:07:07.845402 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:07:07.845355    2572 scope.go:117] "RemoveContainer" containerID="4277ec9534f8745a6c793021229d217d420d7fa63f495ee92c7578e5bcd61a3f"
May 26 12:07:07.853762 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:07:07.853740    2572 scope.go:117] "RemoveContainer" containerID="0c0a4fe7ff814d7f376c002382a4c9fcb920a5b8f93678d68eb0d324da13f820"
May 26 12:07:07.864271 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:07:07.864244    2572 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-5474d9f6f4-kp2g6"]
May 26 12:07:07.867321 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:07:07.867300    2572 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-5474d9f6f4-kp2g6"]
May 26 12:07:09.675578 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:07:09.675543    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9d359422-07c3-419e-aa18-e74f8c38600e" path="/var/lib/kubelet/pods/9d359422-07c3-419e-aa18-e74f8c38600e/volumes"
May 26 12:08:02.729282 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:08:02.729238    2572 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-788b56f4c82qtm2"]
May 26 12:08:02.729874 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:08:02.729627    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d359422-07c3-419e-aa18-e74f8c38600e" containerName="storage-initializer"
May 26 12:08:02.729874 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:08:02.729640    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d359422-07c3-419e-aa18-e74f8c38600e" containerName="storage-initializer"
May 26 12:08:02.729874 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:08:02.729669    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0df0deec-ab10-4a22-95d5-e9d972bf5ea3" containerName="storage-initializer"
May 26 12:08:02.729874 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:08:02.729675    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="0df0deec-ab10-4a22-95d5-e9d972bf5ea3" containerName="storage-initializer"
May 26 12:08:02.729874 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:08:02.729683    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0df0deec-ab10-4a22-95d5-e9d972bf5ea3" containerName="main"
May 26 12:08:02.729874 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:08:02.729688    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="0df0deec-ab10-4a22-95d5-e9d972bf5ea3" containerName="main"
May 26 12:08:02.729874 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:08:02.729695    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d359422-07c3-419e-aa18-e74f8c38600e" containerName="main"
May 26 12:08:02.729874 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:08:02.729700    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d359422-07c3-419e-aa18-e74f8c38600e" containerName="main"
May 26 12:08:02.729874 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:08:02.729815    2572 memory_manager.go:356] "RemoveStaleState removing state" podUID="9d359422-07c3-419e-aa18-e74f8c38600e" containerName="main"
May 26 12:08:02.729874 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:08:02.729828    2572 memory_manager.go:356] "RemoveStaleState removing state" podUID="0df0deec-ab10-4a22-95d5-e9d972bf5ea3" containerName="main"
May 26 12:08:02.731971 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:08:02.731955    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-788b56f4c82qtm2"
May 26 12:08:02.734467 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:08:02.734445    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv8de1d74aab16d9cabd8b5aafeb5248e8-kserve-self-signed-certs\""
May 26 12:08:02.742788 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:08:02.742764    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-788b56f4c82qtm2"]
May 26 12:08:02.917574 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:08:02.917531    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/179716fc-4dad-4baf-b4f1-339ab2bcabf1-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-788b56f4c82qtm2\" (UID: \"179716fc-4dad-4baf-b4f1-339ab2bcabf1\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-788b56f4c82qtm2"
May 26 12:08:02.917574 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:08:02.917580    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9lfxk\" (UniqueName: \"kubernetes.io/projected/179716fc-4dad-4baf-b4f1-339ab2bcabf1-kube-api-access-9lfxk\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-788b56f4c82qtm2\" (UID: \"179716fc-4dad-4baf-b4f1-339ab2bcabf1\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-788b56f4c82qtm2"
May 26 12:08:02.917810 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:08:02.917602    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/179716fc-4dad-4baf-b4f1-339ab2bcabf1-home\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-788b56f4c82qtm2\" (UID: \"179716fc-4dad-4baf-b4f1-339ab2bcabf1\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-788b56f4c82qtm2"
May 26 12:08:02.917810 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:08:02.917626    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/179716fc-4dad-4baf-b4f1-339ab2bcabf1-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-788b56f4c82qtm2\" (UID: \"179716fc-4dad-4baf-b4f1-339ab2bcabf1\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-788b56f4c82qtm2"
May 26 12:08:02.917810 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:08:02.917648    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/179716fc-4dad-4baf-b4f1-339ab2bcabf1-dshm\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-788b56f4c82qtm2\" (UID: \"179716fc-4dad-4baf-b4f1-339ab2bcabf1\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-788b56f4c82qtm2"
May 26 12:08:02.917810 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:08:02.917724    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/179716fc-4dad-4baf-b4f1-339ab2bcabf1-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-788b56f4c82qtm2\" (UID: \"179716fc-4dad-4baf-b4f1-339ab2bcabf1\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-788b56f4c82qtm2"
May 26 12:08:02.917810 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:08:02.917756    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/179716fc-4dad-4baf-b4f1-339ab2bcabf1-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-788b56f4c82qtm2\" (UID: \"179716fc-4dad-4baf-b4f1-339ab2bcabf1\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-788b56f4c82qtm2"
May 26 12:08:03.019163 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:08:03.019060    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/179716fc-4dad-4baf-b4f1-339ab2bcabf1-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-788b56f4c82qtm2\" (UID: \"179716fc-4dad-4baf-b4f1-339ab2bcabf1\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-788b56f4c82qtm2"
May 26 12:08:03.019163 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:08:03.019125    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-9lfxk\" (UniqueName: \"kubernetes.io/projected/179716fc-4dad-4baf-b4f1-339ab2bcabf1-kube-api-access-9lfxk\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-788b56f4c82qtm2\" (UID: \"179716fc-4dad-4baf-b4f1-339ab2bcabf1\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-788b56f4c82qtm2"
May 26 12:08:03.019426 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:08:03.019218    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/179716fc-4dad-4baf-b4f1-339ab2bcabf1-home\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-788b56f4c82qtm2\" (UID: \"179716fc-4dad-4baf-b4f1-339ab2bcabf1\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-788b56f4c82qtm2"
May 26 12:08:03.019426 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:08:03.019263    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/179716fc-4dad-4baf-b4f1-339ab2bcabf1-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-788b56f4c82qtm2\" (UID: \"179716fc-4dad-4baf-b4f1-339ab2bcabf1\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-788b56f4c82qtm2"
May 26 12:08:03.019426 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:08:03.019304    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/179716fc-4dad-4baf-b4f1-339ab2bcabf1-dshm\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-788b56f4c82qtm2\" (UID: \"179716fc-4dad-4baf-b4f1-339ab2bcabf1\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-788b56f4c82qtm2"
May 26 12:08:03.019426 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:08:03.019351    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/179716fc-4dad-4baf-b4f1-339ab2bcabf1-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-788b56f4c82qtm2\" (UID: \"179716fc-4dad-4baf-b4f1-339ab2bcabf1\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-788b56f4c82qtm2"
May 26 12:08:03.019426 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:08:03.019383    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/179716fc-4dad-4baf-b4f1-339ab2bcabf1-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-788b56f4c82qtm2\" (UID: \"179716fc-4dad-4baf-b4f1-339ab2bcabf1\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-788b56f4c82qtm2"
May 26 12:08:03.019696 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:08:03.019605    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/179716fc-4dad-4baf-b4f1-339ab2bcabf1-home\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-788b56f4c82qtm2\" (UID: \"179716fc-4dad-4baf-b4f1-339ab2bcabf1\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-788b56f4c82qtm2"
May 26 12:08:03.019696 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:08:03.019616    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/179716fc-4dad-4baf-b4f1-339ab2bcabf1-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-788b56f4c82qtm2\" (UID: \"179716fc-4dad-4baf-b4f1-339ab2bcabf1\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-788b56f4c82qtm2"
May 26 12:08:03.019696 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:08:03.019667    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/179716fc-4dad-4baf-b4f1-339ab2bcabf1-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-788b56f4c82qtm2\" (UID: \"179716fc-4dad-4baf-b4f1-339ab2bcabf1\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-788b56f4c82qtm2"
May 26 12:08:03.019861 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:08:03.019800    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/179716fc-4dad-4baf-b4f1-339ab2bcabf1-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-788b56f4c82qtm2\" (UID: \"179716fc-4dad-4baf-b4f1-339ab2bcabf1\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-788b56f4c82qtm2"
May 26 12:08:03.021631 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:08:03.021603    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/179716fc-4dad-4baf-b4f1-339ab2bcabf1-dshm\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-788b56f4c82qtm2\" (UID: \"179716fc-4dad-4baf-b4f1-339ab2bcabf1\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-788b56f4c82qtm2"
May 26 12:08:03.021807 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:08:03.021787    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/179716fc-4dad-4baf-b4f1-339ab2bcabf1-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-788b56f4c82qtm2\" (UID: \"179716fc-4dad-4baf-b4f1-339ab2bcabf1\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-788b56f4c82qtm2"
May 26 12:08:03.029629 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:08:03.029606    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9lfxk\" (UniqueName: \"kubernetes.io/projected/179716fc-4dad-4baf-b4f1-339ab2bcabf1-kube-api-access-9lfxk\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-788b56f4c82qtm2\" (UID: \"179716fc-4dad-4baf-b4f1-339ab2bcabf1\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-788b56f4c82qtm2"
May 26 12:08:03.043341 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:08:03.043320    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-788b56f4c82qtm2"
May 26 12:08:03.174991 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:08:03.174964    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-788b56f4c82qtm2"]
May 26 12:08:03.177320 ip-10-0-136-194 kubenswrapper[2572]: W0526 12:08:03.177282    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod179716fc_4dad_4baf_b4f1_339ab2bcabf1.slice/crio-a0c4688f8ed25d7de443606f27e0595cf9ec6371a6ca5388bfa40a2506febd71 WatchSource:0}: Error finding container a0c4688f8ed25d7de443606f27e0595cf9ec6371a6ca5388bfa40a2506febd71: Status 404 returned error can't find the container with id a0c4688f8ed25d7de443606f27e0595cf9ec6371a6ca5388bfa40a2506febd71
May 26 12:08:03.179057 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:08:03.179035    2572 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 26 12:08:04.053656 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:08:04.053623    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-788b56f4c82qtm2" event={"ID":"179716fc-4dad-4baf-b4f1-339ab2bcabf1","Type":"ContainerStarted","Data":"08b729e46667c3c98c0fdb9ac2687f3956a88f871337a135de2e159e3ed5cb6f"}
May 26 12:08:04.054044 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:08:04.053662    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-788b56f4c82qtm2" event={"ID":"179716fc-4dad-4baf-b4f1-339ab2bcabf1","Type":"ContainerStarted","Data":"a0c4688f8ed25d7de443606f27e0595cf9ec6371a6ca5388bfa40a2506febd71"}
May 26 12:08:58.264517 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:08:58.264484    2572 generic.go:358] "Generic (PLEG): container finished" podID="179716fc-4dad-4baf-b4f1-339ab2bcabf1" containerID="08b729e46667c3c98c0fdb9ac2687f3956a88f871337a135de2e159e3ed5cb6f" exitCode=0
May 26 12:08:58.264936 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:08:58.264563    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-788b56f4c82qtm2" event={"ID":"179716fc-4dad-4baf-b4f1-339ab2bcabf1","Type":"ContainerDied","Data":"08b729e46667c3c98c0fdb9ac2687f3956a88f871337a135de2e159e3ed5cb6f"}
May 26 12:08:59.270311 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:08:59.270266    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-788b56f4c82qtm2" event={"ID":"179716fc-4dad-4baf-b4f1-339ab2bcabf1","Type":"ContainerStarted","Data":"f2b4359bd08724847e0482da1ee4ce73a6e1f34766293881ce5ec2e1f4caaf3a"}
May 26 12:08:59.295906 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:08:59.295837    2572 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-788b56f4c82qtm2" podStartSLOduration=57.295820057 podStartE2EDuration="57.295820057s" podCreationTimestamp="2026-05-26 12:08:02 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 12:08:59.293373752 +0000 UTC m=+2662.195584082" watchObservedRunningTime="2026-05-26 12:08:59.295820057 +0000 UTC m=+2662.198030386"
May 26 12:09:03.043842 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:09:03.043805    2572 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-788b56f4c82qtm2"
May 26 12:09:03.044263 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:09:03.043887    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-788b56f4c82qtm2"
May 26 12:09:03.045623 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:09:03.045594    2572 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-788b56f4c82qtm2" podUID="179716fc-4dad-4baf-b4f1-339ab2bcabf1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": dial tcp 10.132.0.51:8000: connect: connection refused"
May 26 12:09:13.044382 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:09:13.044340    2572 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-788b56f4c82qtm2" podUID="179716fc-4dad-4baf-b4f1-339ab2bcabf1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": dial tcp 10.132.0.51:8000: connect: connection refused"
May 26 12:09:23.044584 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:09:23.044491    2572 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-788b56f4c82qtm2" podUID="179716fc-4dad-4baf-b4f1-339ab2bcabf1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": dial tcp 10.132.0.51:8000: connect: connection refused"
May 26 12:09:33.043756 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:09:33.043718    2572 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-788b56f4c82qtm2" podUID="179716fc-4dad-4baf-b4f1-339ab2bcabf1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": dial tcp 10.132.0.51:8000: connect: connection refused"
May 26 12:09:37.821392 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:09:37.821362    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-5bfgg_c6373342-dd07-4497-a4f3-978ea9613b6e/console-operator/2.log"
May 26 12:09:37.827816 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:09:37.827792    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-js8cc_7264c907-f536-4719-905a-2bbda9e3c336/ovn-acl-logging/0.log"
May 26 12:09:37.828712 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:09:37.828689    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-5bfgg_c6373342-dd07-4497-a4f3-978ea9613b6e/console-operator/2.log"
May 26 12:09:37.834748 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:09:37.834729    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-js8cc_7264c907-f536-4719-905a-2bbda9e3c336/ovn-acl-logging/0.log"
May 26 12:09:43.043969 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:09:43.043922    2572 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-788b56f4c82qtm2" podUID="179716fc-4dad-4baf-b4f1-339ab2bcabf1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": dial tcp 10.132.0.51:8000: connect: connection refused"
May 26 12:09:53.044691 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:09:53.044646    2572 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-788b56f4c82qtm2" podUID="179716fc-4dad-4baf-b4f1-339ab2bcabf1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": dial tcp 10.132.0.51:8000: connect: connection refused"
May 26 12:10:03.044337 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:10:03.044293    2572 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-788b56f4c82qtm2" podUID="179716fc-4dad-4baf-b4f1-339ab2bcabf1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": dial tcp 10.132.0.51:8000: connect: connection refused"
May 26 12:10:13.044015 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:10:13.043972    2572 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-788b56f4c82qtm2" podUID="179716fc-4dad-4baf-b4f1-339ab2bcabf1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": dial tcp 10.132.0.51:8000: connect: connection refused"
May 26 12:10:16.236081 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:10:16.236043    2572 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-8578d996bc-4f4n5"]
May 26 12:10:16.236545 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:10:16.236319    2572 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-8578d996bc-4f4n5" podUID="8f0068a6-2aad-417c-a347-14a7cbcc4bf5" containerName="storage-initializer" containerID="cri-o://8cc6b6b19864d9c1319126c6ad3a317501d1307610deac6bbe4de45eed4b5ace" gracePeriod=30
May 26 12:10:16.238980 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:10:16.238950    2572 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7955f5555b-4jdzr"]
May 26 12:10:16.239393 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:10:16.239345    2572 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7955f5555b-4jdzr" podUID="9345be29-4f25-41c5-8489-c02aa4b7cea6" containerName="main" containerID="cri-o://04fc740e951e25f21e8b7cfa3f8c9cdad7a5d6245627a0f74594cb14297fe6b5" gracePeriod=30
May 26 12:10:23.053107 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:10:23.053064    2572 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-788b56f4c82qtm2" podUID="179716fc-4dad-4baf-b4f1-339ab2bcabf1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": dial tcp 10.132.0.51:8000: connect: connection refused"
May 26 12:10:33.054027 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:10:33.053994    2572 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-788b56f4c82qtm2"
May 26 12:10:33.061504 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:10:33.061481    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-788b56f4c82qtm2"
May 26 12:10:39.336936 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:10:39.336889    2572 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5f7f8c5848-99qts"]
May 26 12:10:39.340680 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:10:39.340660    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5f7f8c5848-99qts"
May 26 12:10:39.343403 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:10:39.343377    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-with-refs-pd-test-kserve-self-signed-certs\""
May 26 12:10:39.343545 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:10:39.343519    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-with-refs-pd-test-kserve-dockercfg-crhm9\""
May 26 12:10:39.349356 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:10:39.349335    2572 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7d45d77775-v9cpf"]
May 26 12:10:39.352978 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:10:39.352958    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7d45d77775-v9cpf"
May 26 12:10:39.357319 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:10:39.357298    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5f7f8c5848-99qts"]
May 26 12:10:39.366519 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:10:39.366487    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7d45d77775-v9cpf"]
May 26 12:10:39.411685 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:10:39.411651    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b11682a-156c-4b51-a884-c0be4d37e2ec-tmp-dir\") pod \"router-with-refs-pd-test-kserve-5f7f8c5848-99qts\" (UID: \"9b11682a-156c-4b51-a884-c0be4d37e2ec\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5f7f8c5848-99qts"
May 26 12:10:39.411885 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:10:39.411698    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b11682a-156c-4b51-a884-c0be4d37e2ec-home\") pod \"router-with-refs-pd-test-kserve-5f7f8c5848-99qts\" (UID: \"9b11682a-156c-4b51-a884-c0be4d37e2ec\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5f7f8c5848-99qts"
May 26 12:10:39.411885 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:10:39.411725    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/119a5f43-162c-4430-9b97-ce0cd2c6be9f-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-7d45d77775-v9cpf\" (UID: \"119a5f43-162c-4430-9b97-ce0cd2c6be9f\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7d45d77775-v9cpf"
May 26 12:10:39.412004 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:10:39.411872    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/119a5f43-162c-4430-9b97-ce0cd2c6be9f-home\") pod \"router-with-refs-pd-test-kserve-prefill-7d45d77775-v9cpf\" (UID: \"119a5f43-162c-4430-9b97-ce0cd2c6be9f\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7d45d77775-v9cpf"
May 26 12:10:39.412004 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:10:39.411918    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b11682a-156c-4b51-a884-c0be4d37e2ec-model-cache\") pod \"router-with-refs-pd-test-kserve-5f7f8c5848-99qts\" (UID: \"9b11682a-156c-4b51-a884-c0be4d37e2ec\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5f7f8c5848-99qts"
May 26 12:10:39.412004 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:10:39.411968    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/119a5f43-162c-4430-9b97-ce0cd2c6be9f-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-7d45d77775-v9cpf\" (UID: \"119a5f43-162c-4430-9b97-ce0cd2c6be9f\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7d45d77775-v9cpf"
May 26 12:10:39.412167 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:10:39.412001    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b11682a-156c-4b51-a884-c0be4d37e2ec-tls-certs\") pod \"router-with-refs-pd-test-kserve-5f7f8c5848-99qts\" (UID: \"9b11682a-156c-4b51-a884-c0be4d37e2ec\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5f7f8c5848-99qts"
May 26 12:10:39.412167 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:10:39.412038    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/119a5f43-162c-4430-9b97-ce0cd2c6be9f-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-7d45d77775-v9cpf\" (UID: \"119a5f43-162c-4430-9b97-ce0cd2c6be9f\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7d45d77775-v9cpf"
May 26 12:10:39.412167 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:10:39.412071    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59s69\" (UniqueName: \"kubernetes.io/projected/9b11682a-156c-4b51-a884-c0be4d37e2ec-kube-api-access-59s69\") pod \"router-with-refs-pd-test-kserve-5f7f8c5848-99qts\" (UID: \"9b11682a-156c-4b51-a884-c0be4d37e2ec\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5f7f8c5848-99qts"
May 26 12:10:39.412167 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:10:39.412123    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/119a5f43-162c-4430-9b97-ce0cd2c6be9f-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-7d45d77775-v9cpf\" (UID: \"119a5f43-162c-4430-9b97-ce0cd2c6be9f\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7d45d77775-v9cpf"
May 26 12:10:39.412167 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:10:39.412148    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmztk\" (UniqueName: \"kubernetes.io/projected/119a5f43-162c-4430-9b97-ce0cd2c6be9f-kube-api-access-bmztk\") pod \"router-with-refs-pd-test-kserve-prefill-7d45d77775-v9cpf\" (UID: \"119a5f43-162c-4430-9b97-ce0cd2c6be9f\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7d45d77775-v9cpf"
May 26 12:10:39.412458 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:10:39.412174    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b11682a-156c-4b51-a884-c0be4d37e2ec-dshm\") pod \"router-with-refs-pd-test-kserve-5f7f8c5848-99qts\" (UID: \"9b11682a-156c-4b51-a884-c0be4d37e2ec\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5f7f8c5848-99qts"
May 26 12:10:39.412458 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:10:39.412302    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9b11682a-156c-4b51-a884-c0be4d37e2ec-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-5f7f8c5848-99qts\" (UID: \"9b11682a-156c-4b51-a884-c0be4d37e2ec\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5f7f8c5848-99qts"
May 26 12:10:39.412458 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:10:39.412339    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/119a5f43-162c-4430-9b97-ce0cd2c6be9f-tmp-dir\") pod \"router-with-refs-pd-test-kserve-prefill-7d45d77775-v9cpf\" (UID: \"119a5f43-162c-4430-9b97-ce0cd2c6be9f\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7d45d77775-v9cpf"
May 26 12:10:39.513381 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:10:39.513345    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/119a5f43-162c-4430-9b97-ce0cd2c6be9f-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-7d45d77775-v9cpf\" (UID: \"119a5f43-162c-4430-9b97-ce0cd2c6be9f\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7d45d77775-v9cpf"
May 26 12:10:39.513381 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:10:39.513383    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-bmztk\" (UniqueName: \"kubernetes.io/projected/119a5f43-162c-4430-9b97-ce0cd2c6be9f-kube-api-access-bmztk\") pod \"router-with-refs-pd-test-kserve-prefill-7d45d77775-v9cpf\" (UID: \"119a5f43-162c-4430-9b97-ce0cd2c6be9f\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7d45d77775-v9cpf"
May 26 12:10:39.513638 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:10:39.513403    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b11682a-156c-4b51-a884-c0be4d37e2ec-dshm\") pod \"router-with-refs-pd-test-kserve-5f7f8c5848-99qts\" (UID: \"9b11682a-156c-4b51-a884-c0be4d37e2ec\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5f7f8c5848-99qts"
May 26 12:10:39.513638 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:10:39.513451    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9b11682a-156c-4b51-a884-c0be4d37e2ec-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-5f7f8c5848-99qts\" (UID: \"9b11682a-156c-4b51-a884-c0be4d37e2ec\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5f7f8c5848-99qts"
May 26 12:10:39.513638 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:10:39.513478    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/119a5f43-162c-4430-9b97-ce0cd2c6be9f-tmp-dir\") pod \"router-with-refs-pd-test-kserve-prefill-7d45d77775-v9cpf\" (UID: \"119a5f43-162c-4430-9b97-ce0cd2c6be9f\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7d45d77775-v9cpf"
May 26 12:10:39.513638 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:10:39.513521    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b11682a-156c-4b51-a884-c0be4d37e2ec-tmp-dir\") pod \"router-with-refs-pd-test-kserve-5f7f8c5848-99qts\" (UID: \"9b11682a-156c-4b51-a884-c0be4d37e2ec\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5f7f8c5848-99qts"
May 26 12:10:39.513638 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:10:39.513549    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b11682a-156c-4b51-a884-c0be4d37e2ec-home\") pod \"router-with-refs-pd-test-kserve-5f7f8c5848-99qts\" (UID: \"9b11682a-156c-4b51-a884-c0be4d37e2ec\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5f7f8c5848-99qts"
May 26 12:10:39.513638 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:10:39.513576    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/119a5f43-162c-4430-9b97-ce0cd2c6be9f-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-7d45d77775-v9cpf\" (UID: \"119a5f43-162c-4430-9b97-ce0cd2c6be9f\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7d45d77775-v9cpf"
May 26 12:10:39.513638 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:10:39.513628    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/119a5f43-162c-4430-9b97-ce0cd2c6be9f-home\") pod \"router-with-refs-pd-test-kserve-prefill-7d45d77775-v9cpf\" (UID: \"119a5f43-162c-4430-9b97-ce0cd2c6be9f\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7d45d77775-v9cpf"
May 26 12:10:39.514011 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:10:39.513657    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b11682a-156c-4b51-a884-c0be4d37e2ec-model-cache\") pod \"router-with-refs-pd-test-kserve-5f7f8c5848-99qts\" (UID: \"9b11682a-156c-4b51-a884-c0be4d37e2ec\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5f7f8c5848-99qts"
May 26 12:10:39.514011 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:10:39.513692    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/119a5f43-162c-4430-9b97-ce0cd2c6be9f-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-7d45d77775-v9cpf\" (UID: \"119a5f43-162c-4430-9b97-ce0cd2c6be9f\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7d45d77775-v9cpf"
May 26 12:10:39.514011 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:10:39.513716    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b11682a-156c-4b51-a884-c0be4d37e2ec-tls-certs\") pod \"router-with-refs-pd-test-kserve-5f7f8c5848-99qts\" (UID: \"9b11682a-156c-4b51-a884-c0be4d37e2ec\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5f7f8c5848-99qts"
May 26 12:10:39.514011 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:10:39.513742    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/119a5f43-162c-4430-9b97-ce0cd2c6be9f-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-7d45d77775-v9cpf\" (UID: \"119a5f43-162c-4430-9b97-ce0cd2c6be9f\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7d45d77775-v9cpf"
May 26 12:10:39.514011 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:10:39.513796    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-59s69\" (UniqueName: \"kubernetes.io/projected/9b11682a-156c-4b51-a884-c0be4d37e2ec-kube-api-access-59s69\") pod \"router-with-refs-pd-test-kserve-5f7f8c5848-99qts\" (UID: \"9b11682a-156c-4b51-a884-c0be4d37e2ec\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5f7f8c5848-99qts"
May 26 12:10:39.514011 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:10:39.513869    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/119a5f43-162c-4430-9b97-ce0cd2c6be9f-tmp-dir\") pod \"router-with-refs-pd-test-kserve-prefill-7d45d77775-v9cpf\" (UID: \"119a5f43-162c-4430-9b97-ce0cd2c6be9f\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7d45d77775-v9cpf"
May 26 12:10:39.514011 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:10:39.513925    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b11682a-156c-4b51-a884-c0be4d37e2ec-tmp-dir\") pod \"router-with-refs-pd-test-kserve-5f7f8c5848-99qts\" (UID: \"9b11682a-156c-4b51-a884-c0be4d37e2ec\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5f7f8c5848-99qts"
May 26 12:10:39.514011 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:10:39.513965    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b11682a-156c-4b51-a884-c0be4d37e2ec-home\") pod \"router-with-refs-pd-test-kserve-5f7f8c5848-99qts\" (UID: \"9b11682a-156c-4b51-a884-c0be4d37e2ec\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5f7f8c5848-99qts"
May 26 12:10:39.514671 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:10:39.514031    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b11682a-156c-4b51-a884-c0be4d37e2ec-model-cache\") pod \"router-with-refs-pd-test-kserve-5f7f8c5848-99qts\" (UID: \"9b11682a-156c-4b51-a884-c0be4d37e2ec\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5f7f8c5848-99qts"
May 26 12:10:39.514671 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:10:39.514183    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/119a5f43-162c-4430-9b97-ce0cd2c6be9f-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-7d45d77775-v9cpf\" (UID: \"119a5f43-162c-4430-9b97-ce0cd2c6be9f\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7d45d77775-v9cpf"
May 26 12:10:39.514671 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:10:39.514389    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9b11682a-156c-4b51-a884-c0be4d37e2ec-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-5f7f8c5848-99qts\" (UID: \"9b11682a-156c-4b51-a884-c0be4d37e2ec\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5f7f8c5848-99qts"
May 26 12:10:39.514671 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:10:39.514457    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/119a5f43-162c-4430-9b97-ce0cd2c6be9f-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-7d45d77775-v9cpf\" (UID: \"119a5f43-162c-4430-9b97-ce0cd2c6be9f\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7d45d77775-v9cpf"
May 26 12:10:39.514671 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:10:39.514585    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/119a5f43-162c-4430-9b97-ce0cd2c6be9f-home\") pod \"router-with-refs-pd-test-kserve-prefill-7d45d77775-v9cpf\" (UID: \"119a5f43-162c-4430-9b97-ce0cd2c6be9f\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7d45d77775-v9cpf"
May 26 12:10:39.516281 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:10:39.516247    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b11682a-156c-4b51-a884-c0be4d37e2ec-dshm\") pod \"router-with-refs-pd-test-kserve-5f7f8c5848-99qts\" (UID: \"9b11682a-156c-4b51-a884-c0be4d37e2ec\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5f7f8c5848-99qts"
May 26 12:10:39.516443 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:10:39.516424    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/119a5f43-162c-4430-9b97-ce0cd2c6be9f-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-7d45d77775-v9cpf\" (UID: \"119a5f43-162c-4430-9b97-ce0cd2c6be9f\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7d45d77775-v9cpf"
May 26 12:10:39.516800 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:10:39.516784    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b11682a-156c-4b51-a884-c0be4d37e2ec-tls-certs\") pod \"router-with-refs-pd-test-kserve-5f7f8c5848-99qts\" (UID: \"9b11682a-156c-4b51-a884-c0be4d37e2ec\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5f7f8c5848-99qts"
May 26 12:10:39.516925 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:10:39.516901    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/119a5f43-162c-4430-9b97-ce0cd2c6be9f-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-7d45d77775-v9cpf\" (UID: \"119a5f43-162c-4430-9b97-ce0cd2c6be9f\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7d45d77775-v9cpf"
May 26 12:10:39.521615 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:10:39.521590    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-59s69\" (UniqueName: \"kubernetes.io/projected/9b11682a-156c-4b51-a884-c0be4d37e2ec-kube-api-access-59s69\") pod \"router-with-refs-pd-test-kserve-5f7f8c5848-99qts\" (UID: \"9b11682a-156c-4b51-a884-c0be4d37e2ec\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5f7f8c5848-99qts"
May 26 12:10:39.521707 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:10:39.521675    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmztk\" (UniqueName: \"kubernetes.io/projected/119a5f43-162c-4430-9b97-ce0cd2c6be9f-kube-api-access-bmztk\") pod \"router-with-refs-pd-test-kserve-prefill-7d45d77775-v9cpf\" (UID: \"119a5f43-162c-4430-9b97-ce0cd2c6be9f\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7d45d77775-v9cpf"
May 26 12:10:39.652433 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:10:39.652347    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5f7f8c5848-99qts"
May 26 12:10:39.665951 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:10:39.665928    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7d45d77775-v9cpf"
May 26 12:10:39.821758 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:10:39.821729    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5f7f8c5848-99qts"]
May 26 12:10:39.824673 ip-10-0-136-194 kubenswrapper[2572]: W0526 12:10:39.824642    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b11682a_156c_4b51_a884_c0be4d37e2ec.slice/crio-29e2e18ab9411fee37ff6859f89664eac2ad0324fc8f06966052342838615dec WatchSource:0}: Error finding container 29e2e18ab9411fee37ff6859f89664eac2ad0324fc8f06966052342838615dec: Status 404 returned error can't find the container with id 29e2e18ab9411fee37ff6859f89664eac2ad0324fc8f06966052342838615dec
May 26 12:10:39.833370 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:10:39.833334    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7d45d77775-v9cpf"]
May 26 12:10:39.837393 ip-10-0-136-194 kubenswrapper[2572]: W0526 12:10:39.837370    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod119a5f43_162c_4430_9b97_ce0cd2c6be9f.slice/crio-7daf59a429bb38c9a0effbdde1e94ec5fe2bc8472b06f51dbabdf56d9f3b2bff WatchSource:0}: Error finding container 7daf59a429bb38c9a0effbdde1e94ec5fe2bc8472b06f51dbabdf56d9f3b2bff: Status 404 returned error can't find the container with id 7daf59a429bb38c9a0effbdde1e94ec5fe2bc8472b06f51dbabdf56d9f3b2bff
May 26 12:10:40.638120 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:10:40.638069    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5f7f8c5848-99qts" event={"ID":"9b11682a-156c-4b51-a884-c0be4d37e2ec","Type":"ContainerStarted","Data":"7daef1c0674dbcd8bff1d5c55f6fce7d309cf4243214b77537f86599a2c6ba5c"}
May 26 12:10:40.638120 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:10:40.638116    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5f7f8c5848-99qts" event={"ID":"9b11682a-156c-4b51-a884-c0be4d37e2ec","Type":"ContainerStarted","Data":"29e2e18ab9411fee37ff6859f89664eac2ad0324fc8f06966052342838615dec"}
May 26 12:10:40.638639 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:10:40.638222    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5f7f8c5848-99qts"
May 26 12:10:40.639473 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:10:40.639444    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7d45d77775-v9cpf" event={"ID":"119a5f43-162c-4430-9b97-ce0cd2c6be9f","Type":"ContainerStarted","Data":"daad443a8d340449e2dd4d8d794ef079e69fe6e0a2ade5d325953a0b50132a56"}
May 26 12:10:40.639473 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:10:40.639474    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7d45d77775-v9cpf" event={"ID":"119a5f43-162c-4430-9b97-ce0cd2c6be9f","Type":"ContainerStarted","Data":"7daf59a429bb38c9a0effbdde1e94ec5fe2bc8472b06f51dbabdf56d9f3b2bff"}
May 26 12:10:41.645746 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:10:41.645708    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5f7f8c5848-99qts" event={"ID":"9b11682a-156c-4b51-a884-c0be4d37e2ec","Type":"ContainerStarted","Data":"233838b208a78960bbb35472a1780e222f89b843f401fd11c83093b559751d55"}
May 26 12:10:45.665391 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:10:45.665346    2572 generic.go:358] "Generic (PLEG): container finished" podID="9b11682a-156c-4b51-a884-c0be4d37e2ec" containerID="233838b208a78960bbb35472a1780e222f89b843f401fd11c83093b559751d55" exitCode=0
May 26 12:10:45.665951 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:10:45.665400    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5f7f8c5848-99qts" event={"ID":"9b11682a-156c-4b51-a884-c0be4d37e2ec","Type":"ContainerDied","Data":"233838b208a78960bbb35472a1780e222f89b843f401fd11c83093b559751d55"}
May 26 12:10:46.240354 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:10:46.240297    2572 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7955f5555b-4jdzr" podUID="9345be29-4f25-41c5-8489-c02aa4b7cea6" containerName="llm-d-routing-sidecar" containerID="cri-o://6b7575685b8f04d1ea4f8b73617a67d16d911c9eb98929e3f5827bfa1b255dfe" gracePeriod=2
May 26 12:10:46.537958 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:10:46.537934    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_custom-route-timeout-pd-test-kserve-7955f5555b-4jdzr_9345be29-4f25-41c5-8489-c02aa4b7cea6/main/0.log"
May 26 12:10:46.538660 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:10:46.538644    2572 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7955f5555b-4jdzr"
May 26 12:10:46.671882 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:10:46.671847    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5f7f8c5848-99qts" event={"ID":"9b11682a-156c-4b51-a884-c0be4d37e2ec","Type":"ContainerStarted","Data":"70138a75f3c3d1a2cc0a1cc5bdb1a80d705880a5acbd87e5975f0c6d6231cfe3"}
May 26 12:10:46.673151 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:10:46.673131    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_custom-route-timeout-pd-test-kserve-7955f5555b-4jdzr_9345be29-4f25-41c5-8489-c02aa4b7cea6/main/0.log"
May 26 12:10:46.673748 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:10:46.673725    2572 generic.go:358] "Generic (PLEG): container finished" podID="9345be29-4f25-41c5-8489-c02aa4b7cea6" containerID="04fc740e951e25f21e8b7cfa3f8c9cdad7a5d6245627a0f74594cb14297fe6b5" exitCode=137
May 26 12:10:46.673830 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:10:46.673750    2572 generic.go:358] "Generic (PLEG): container finished" podID="9345be29-4f25-41c5-8489-c02aa4b7cea6" containerID="6b7575685b8f04d1ea4f8b73617a67d16d911c9eb98929e3f5827bfa1b255dfe" exitCode=0
May 26 12:10:46.673830 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:10:46.673791    2572 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7955f5555b-4jdzr"
May 26 12:10:46.673939 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:10:46.673785    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7955f5555b-4jdzr" event={"ID":"9345be29-4f25-41c5-8489-c02aa4b7cea6","Type":"ContainerDied","Data":"04fc740e951e25f21e8b7cfa3f8c9cdad7a5d6245627a0f74594cb14297fe6b5"}
May 26 12:10:46.673939 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:10:46.673905    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7955f5555b-4jdzr" event={"ID":"9345be29-4f25-41c5-8489-c02aa4b7cea6","Type":"ContainerDied","Data":"6b7575685b8f04d1ea4f8b73617a67d16d911c9eb98929e3f5827bfa1b255dfe"}
May 26 12:10:46.673939 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:10:46.673935    2572 scope.go:117] "RemoveContainer" containerID="04fc740e951e25f21e8b7cfa3f8c9cdad7a5d6245627a0f74594cb14297fe6b5"
May 26 12:10:46.674100 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:10:46.673939    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7955f5555b-4jdzr" event={"ID":"9345be29-4f25-41c5-8489-c02aa4b7cea6","Type":"ContainerDied","Data":"983251a9cde037b28e39326ec2593a0f10b245f33566e05f51415c7b805efdcf"}
May 26 12:10:46.675257 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:10:46.675241    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_custom-route-timeout-pd-test-kserve-prefill-8578d996bc-4f4n5_8f0068a6-2aad-417c-a347-14a7cbcc4bf5/storage-initializer/0.log"
May 26 12:10:46.675348 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:10:46.675276    2572 generic.go:358] "Generic (PLEG): container finished" podID="8f0068a6-2aad-417c-a347-14a7cbcc4bf5" containerID="8cc6b6b19864d9c1319126c6ad3a317501d1307610deac6bbe4de45eed4b5ace" exitCode=137
May 26 12:10:46.675348 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:10:46.675335    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-8578d996bc-4f4n5" event={"ID":"8f0068a6-2aad-417c-a347-14a7cbcc4bf5","Type":"ContainerDied","Data":"8cc6b6b19864d9c1319126c6ad3a317501d1307610deac6bbe4de45eed4b5ace"}
May 26 12:10:46.683418 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:10:46.683400    2572 scope.go:117] "RemoveContainer" containerID="7bf1b026f2a24314b13051d8e3b49573861946c826270ea57ba842dde540ec84"
May 26 12:10:46.697370 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:10:46.697316    2572 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5f7f8c5848-99qts" podStartSLOduration=7.6973008929999995 podStartE2EDuration="7.697300893s" podCreationTimestamp="2026-05-26 12:10:39 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 12:10:46.695004591 +0000 UTC m=+2769.597214921" watchObservedRunningTime="2026-05-26 12:10:46.697300893 +0000 UTC m=+2769.599511223"
May 26 12:10:46.702849 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:10:46.702830    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9345be29-4f25-41c5-8489-c02aa4b7cea6-tls-certs\") pod \"9345be29-4f25-41c5-8489-c02aa4b7cea6\" (UID: \"9345be29-4f25-41c5-8489-c02aa4b7cea6\") "
May 26 12:10:46.702963 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:10:46.702871    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9345be29-4f25-41c5-8489-c02aa4b7cea6-dshm\") pod \"9345be29-4f25-41c5-8489-c02aa4b7cea6\" (UID: \"9345be29-4f25-41c5-8489-c02aa4b7cea6\") "
May 26 12:10:46.702963 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:10:46.702895    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8q865\" (UniqueName: \"kubernetes.io/projected/9345be29-4f25-41c5-8489-c02aa4b7cea6-kube-api-access-8q865\") pod \"9345be29-4f25-41c5-8489-c02aa4b7cea6\" (UID: \"9345be29-4f25-41c5-8489-c02aa4b7cea6\") "
May 26 12:10:46.702963 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:10:46.702918    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9345be29-4f25-41c5-8489-c02aa4b7cea6-home\") pod \"9345be29-4f25-41c5-8489-c02aa4b7cea6\" (UID: \"9345be29-4f25-41c5-8489-c02aa4b7cea6\") "
May 26 12:10:46.703130 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:10:46.703038    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9345be29-4f25-41c5-8489-c02aa4b7cea6-tmp-dir\") pod \"9345be29-4f25-41c5-8489-c02aa4b7cea6\" (UID: \"9345be29-4f25-41c5-8489-c02aa4b7cea6\") "
May 26 12:10:46.703130 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:10:46.703107    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9345be29-4f25-41c5-8489-c02aa4b7cea6-model-cache\") pod \"9345be29-4f25-41c5-8489-c02aa4b7cea6\" (UID: \"9345be29-4f25-41c5-8489-c02aa4b7cea6\") "
May 26 12:10:46.703760 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:10:46.703164    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9345be29-4f25-41c5-8489-c02aa4b7cea6-kserve-provision-location\") pod \"9345be29-4f25-41c5-8489-c02aa4b7cea6\" (UID: \"9345be29-4f25-41c5-8489-c02aa4b7cea6\") "
May 26 12:10:46.703760 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:10:46.703466    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9345be29-4f25-41c5-8489-c02aa4b7cea6-model-cache" (OuterVolumeSpecName: "model-cache") pod "9345be29-4f25-41c5-8489-c02aa4b7cea6" (UID: "9345be29-4f25-41c5-8489-c02aa4b7cea6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 12:10:46.703760 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:10:46.703572    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9345be29-4f25-41c5-8489-c02aa4b7cea6-home" (OuterVolumeSpecName: "home") pod "9345be29-4f25-41c5-8489-c02aa4b7cea6" (UID: "9345be29-4f25-41c5-8489-c02aa4b7cea6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 12:10:46.703760 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:10:46.703597    2572 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9345be29-4f25-41c5-8489-c02aa4b7cea6-model-cache\") on node \"ip-10-0-136-194.ec2.internal\" DevicePath \"\""
May 26 12:10:46.705118 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:10:46.705086    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9345be29-4f25-41c5-8489-c02aa4b7cea6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9345be29-4f25-41c5-8489-c02aa4b7cea6" (UID: "9345be29-4f25-41c5-8489-c02aa4b7cea6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 12:10:46.705450 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:10:46.705429    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9345be29-4f25-41c5-8489-c02aa4b7cea6-dshm" (OuterVolumeSpecName: "dshm") pod "9345be29-4f25-41c5-8489-c02aa4b7cea6" (UID: "9345be29-4f25-41c5-8489-c02aa4b7cea6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 12:10:46.705840 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:10:46.705823    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9345be29-4f25-41c5-8489-c02aa4b7cea6-kube-api-access-8q865" (OuterVolumeSpecName: "kube-api-access-8q865") pod "9345be29-4f25-41c5-8489-c02aa4b7cea6" (UID: "9345be29-4f25-41c5-8489-c02aa4b7cea6"). InnerVolumeSpecName "kube-api-access-8q865". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 12:10:46.717679 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:10:46.717658    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9345be29-4f25-41c5-8489-c02aa4b7cea6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9345be29-4f25-41c5-8489-c02aa4b7cea6" (UID: "9345be29-4f25-41c5-8489-c02aa4b7cea6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 12:10:46.750551 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:10:46.750436    2572 scope.go:117] "RemoveContainer" containerID="6b7575685b8f04d1ea4f8b73617a67d16d911c9eb98929e3f5827bfa1b255dfe"
May 26 12:10:46.759464 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:10:46.759434    2572 scope.go:117] "RemoveContainer" containerID="04fc740e951e25f21e8b7cfa3f8c9cdad7a5d6245627a0f74594cb14297fe6b5"
May 26 12:10:46.759785 ip-10-0-136-194 kubenswrapper[2572]: E0526 12:10:46.759757    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04fc740e951e25f21e8b7cfa3f8c9cdad7a5d6245627a0f74594cb14297fe6b5\": container with ID starting with 04fc740e951e25f21e8b7cfa3f8c9cdad7a5d6245627a0f74594cb14297fe6b5 not found: ID does not exist" containerID="04fc740e951e25f21e8b7cfa3f8c9cdad7a5d6245627a0f74594cb14297fe6b5"
May 26 12:10:46.759900 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:10:46.759797    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04fc740e951e25f21e8b7cfa3f8c9cdad7a5d6245627a0f74594cb14297fe6b5"} err="failed to get container status \"04fc740e951e25f21e8b7cfa3f8c9cdad7a5d6245627a0f74594cb14297fe6b5\": rpc error: code = NotFound desc = could not find container \"04fc740e951e25f21e8b7cfa3f8c9cdad7a5d6245627a0f74594cb14297fe6b5\": container with ID starting with 04fc740e951e25f21e8b7cfa3f8c9cdad7a5d6245627a0f74594cb14297fe6b5 not found: ID does not exist"
May 26 12:10:46.759900 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:10:46.759827    2572 scope.go:117] "RemoveContainer" containerID="7bf1b026f2a24314b13051d8e3b49573861946c826270ea57ba842dde540ec84"
May 26 12:10:46.760169 ip-10-0-136-194 kubenswrapper[2572]: E0526 12:10:46.760142    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7bf1b026f2a24314b13051d8e3b49573861946c826270ea57ba842dde540ec84\": container with ID starting with 7bf1b026f2a24314b13051d8e3b49573861946c826270ea57ba842dde540ec84 not found: ID does not exist" containerID="7bf1b026f2a24314b13051d8e3b49573861946c826270ea57ba842dde540ec84"
May 26 12:10:46.760385 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:10:46.760182    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7bf1b026f2a24314b13051d8e3b49573861946c826270ea57ba842dde540ec84"} err="failed to get container status \"7bf1b026f2a24314b13051d8e3b49573861946c826270ea57ba842dde540ec84\": rpc error: code = NotFound desc = could not find container \"7bf1b026f2a24314b13051d8e3b49573861946c826270ea57ba842dde540ec84\": container with ID starting with 7bf1b026f2a24314b13051d8e3b49573861946c826270ea57ba842dde540ec84 not found: ID does not exist"
May 26 12:10:46.760385 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:10:46.760231    2572 scope.go:117] "RemoveContainer" containerID="6b7575685b8f04d1ea4f8b73617a67d16d911c9eb98929e3f5827bfa1b255dfe"
May 26 12:10:46.760521 ip-10-0-136-194 kubenswrapper[2572]: E0526 12:10:46.760489    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b7575685b8f04d1ea4f8b73617a67d16d911c9eb98929e3f5827bfa1b255dfe\": container with ID starting with 6b7575685b8f04d1ea4f8b73617a67d16d911c9eb98929e3f5827bfa1b255dfe not found: ID does not exist" containerID="6b7575685b8f04d1ea4f8b73617a67d16d911c9eb98929e3f5827bfa1b255dfe"
May 26 12:10:46.760565 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:10:46.760530    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b7575685b8f04d1ea4f8b73617a67d16d911c9eb98929e3f5827bfa1b255dfe"} err="failed to get container status \"6b7575685b8f04d1ea4f8b73617a67d16d911c9eb98929e3f5827bfa1b255dfe\": rpc error: code = NotFound desc = could not find container \"6b7575685b8f04d1ea4f8b73617a67d16d911c9eb98929e3f5827bfa1b255dfe\": container with ID starting with 6b7575685b8f04d1ea4f8b73617a67d16d911c9eb98929e3f5827bfa1b255dfe not found: ID does not exist"
May 26 12:10:46.760565 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:10:46.760553    2572 scope.go:117] "RemoveContainer" containerID="04fc740e951e25f21e8b7cfa3f8c9cdad7a5d6245627a0f74594cb14297fe6b5"
May 26 12:10:46.760861 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:10:46.760840    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04fc740e951e25f21e8b7cfa3f8c9cdad7a5d6245627a0f74594cb14297fe6b5"} err="failed to get container status \"04fc740e951e25f21e8b7cfa3f8c9cdad7a5d6245627a0f74594cb14297fe6b5\": rpc error: code = NotFound desc = could not find container \"04fc740e951e25f21e8b7cfa3f8c9cdad7a5d6245627a0f74594cb14297fe6b5\": container with ID starting with 04fc740e951e25f21e8b7cfa3f8c9cdad7a5d6245627a0f74594cb14297fe6b5 not found: ID does not exist"
May 26 12:10:46.760939 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:10:46.760863    2572 scope.go:117] "RemoveContainer" containerID="7bf1b026f2a24314b13051d8e3b49573861946c826270ea57ba842dde540ec84"
May 26 12:10:46.761157 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:10:46.761134    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7bf1b026f2a24314b13051d8e3b49573861946c826270ea57ba842dde540ec84"} err="failed to get container status \"7bf1b026f2a24314b13051d8e3b49573861946c826270ea57ba842dde540ec84\": rpc error: code = NotFound desc = could not find container \"7bf1b026f2a24314b13051d8e3b49573861946c826270ea57ba842dde540ec84\": container with ID starting with 7bf1b026f2a24314b13051d8e3b49573861946c826270ea57ba842dde540ec84 not found: ID does not exist"
May 26 12:10:46.761224 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:10:46.761169    2572 scope.go:117] "RemoveContainer" containerID="6b7575685b8f04d1ea4f8b73617a67d16d911c9eb98929e3f5827bfa1b255dfe"
May 26 12:10:46.761474 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:10:46.761453    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b7575685b8f04d1ea4f8b73617a67d16d911c9eb98929e3f5827bfa1b255dfe"} err="failed to get container status \"6b7575685b8f04d1ea4f8b73617a67d16d911c9eb98929e3f5827bfa1b255dfe\": rpc error: code = NotFound desc = could not find container \"6b7575685b8f04d1ea4f8b73617a67d16d911c9eb98929e3f5827bfa1b255dfe\": container with ID starting with 6b7575685b8f04d1ea4f8b73617a67d16d911c9eb98929e3f5827bfa1b255dfe not found: ID does not exist"
May 26 12:10:46.768120 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:10:46.768094    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9345be29-4f25-41c5-8489-c02aa4b7cea6-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "9345be29-4f25-41c5-8489-c02aa4b7cea6" (UID: "9345be29-4f25-41c5-8489-c02aa4b7cea6"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 12:10:46.804517 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:10:46.804423    2572 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9345be29-4f25-41c5-8489-c02aa4b7cea6-kserve-provision-location\") on node \"ip-10-0-136-194.ec2.internal\" DevicePath \"\""
May 26 12:10:46.804517 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:10:46.804460    2572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9345be29-4f25-41c5-8489-c02aa4b7cea6-tls-certs\") on node \"ip-10-0-136-194.ec2.internal\" DevicePath \"\""
May 26 12:10:46.804517 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:10:46.804475    2572 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9345be29-4f25-41c5-8489-c02aa4b7cea6-dshm\") on node \"ip-10-0-136-194.ec2.internal\" DevicePath \"\""
May 26 12:10:46.804517 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:10:46.804489    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8q865\" (UniqueName: \"kubernetes.io/projected/9345be29-4f25-41c5-8489-c02aa4b7cea6-kube-api-access-8q865\") on node \"ip-10-0-136-194.ec2.internal\" DevicePath \"\""
May 26 12:10:46.804517 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:10:46.804506    2572 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9345be29-4f25-41c5-8489-c02aa4b7cea6-home\") on node \"ip-10-0-136-194.ec2.internal\" DevicePath \"\""
May 26 12:10:46.804895 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:10:46.804521    2572 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9345be29-4f25-41c5-8489-c02aa4b7cea6-tmp-dir\") on node \"ip-10-0-136-194.ec2.internal\" DevicePath \"\""
May 26 12:10:46.876469 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:10:46.876442    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_custom-route-timeout-pd-test-kserve-prefill-8578d996bc-4f4n5_8f0068a6-2aad-417c-a347-14a7cbcc4bf5/storage-initializer/0.log"
May 26 12:10:46.876627 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:10:46.876526    2572 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-8578d996bc-4f4n5"
May 26 12:10:46.997921 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:10:46.997886    2572 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7955f5555b-4jdzr"]
May 26 12:10:47.001390 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:10:47.001363    2572 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-7955f5555b-4jdzr"]
May 26 12:10:47.006319 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:10:47.006297    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f0068a6-2aad-417c-a347-14a7cbcc4bf5-tmp-dir\") pod \"8f0068a6-2aad-417c-a347-14a7cbcc4bf5\" (UID: \"8f0068a6-2aad-417c-a347-14a7cbcc4bf5\") "
May 26 12:10:47.006432 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:10:47.006340    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f0068a6-2aad-417c-a347-14a7cbcc4bf5-dshm\") pod \"8f0068a6-2aad-417c-a347-14a7cbcc4bf5\" (UID: \"8f0068a6-2aad-417c-a347-14a7cbcc4bf5\") "
May 26 12:10:47.006432 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:10:47.006371    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f0068a6-2aad-417c-a347-14a7cbcc4bf5-home\") pod \"8f0068a6-2aad-417c-a347-14a7cbcc4bf5\" (UID: \"8f0068a6-2aad-417c-a347-14a7cbcc4bf5\") "
May 26 12:10:47.006432 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:10:47.006398    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vvfjc\" (UniqueName: \"kubernetes.io/projected/8f0068a6-2aad-417c-a347-14a7cbcc4bf5-kube-api-access-vvfjc\") pod \"8f0068a6-2aad-417c-a347-14a7cbcc4bf5\" (UID: \"8f0068a6-2aad-417c-a347-14a7cbcc4bf5\") "
May 26 12:10:47.006432 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:10:47.006414    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8f0068a6-2aad-417c-a347-14a7cbcc4bf5-kserve-provision-location\") pod \"8f0068a6-2aad-417c-a347-14a7cbcc4bf5\" (UID: \"8f0068a6-2aad-417c-a347-14a7cbcc4bf5\") "
May 26 12:10:47.006626 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:10:47.006465    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f0068a6-2aad-417c-a347-14a7cbcc4bf5-tls-certs\") pod \"8f0068a6-2aad-417c-a347-14a7cbcc4bf5\" (UID: \"8f0068a6-2aad-417c-a347-14a7cbcc4bf5\") "
May 26 12:10:47.006626 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:10:47.006537    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f0068a6-2aad-417c-a347-14a7cbcc4bf5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8f0068a6-2aad-417c-a347-14a7cbcc4bf5" (UID: "8f0068a6-2aad-417c-a347-14a7cbcc4bf5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 12:10:47.006626 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:10:47.006547    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f0068a6-2aad-417c-a347-14a7cbcc4bf5-model-cache\") pod \"8f0068a6-2aad-417c-a347-14a7cbcc4bf5\" (UID: \"8f0068a6-2aad-417c-a347-14a7cbcc4bf5\") "
May 26 12:10:47.006777 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:10:47.006672    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f0068a6-2aad-417c-a347-14a7cbcc4bf5-home" (OuterVolumeSpecName: "home") pod "8f0068a6-2aad-417c-a347-14a7cbcc4bf5" (UID: "8f0068a6-2aad-417c-a347-14a7cbcc4bf5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 12:10:47.006777 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:10:47.006742    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f0068a6-2aad-417c-a347-14a7cbcc4bf5-model-cache" (OuterVolumeSpecName: "model-cache") pod "8f0068a6-2aad-417c-a347-14a7cbcc4bf5" (UID: "8f0068a6-2aad-417c-a347-14a7cbcc4bf5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 12:10:47.006921 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:10:47.006900    2572 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f0068a6-2aad-417c-a347-14a7cbcc4bf5-model-cache\") on node \"ip-10-0-136-194.ec2.internal\" DevicePath \"\""
May 26 12:10:47.006921 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:10:47.006923    2572 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f0068a6-2aad-417c-a347-14a7cbcc4bf5-tmp-dir\") on node \"ip-10-0-136-194.ec2.internal\" DevicePath \"\""
May 26 12:10:47.007123 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:10:47.006936    2572 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f0068a6-2aad-417c-a347-14a7cbcc4bf5-home\") on node \"ip-10-0-136-194.ec2.internal\" DevicePath \"\""
May 26 12:10:47.008598 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:10:47.008576    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f0068a6-2aad-417c-a347-14a7cbcc4bf5-kube-api-access-vvfjc" (OuterVolumeSpecName: "kube-api-access-vvfjc") pod "8f0068a6-2aad-417c-a347-14a7cbcc4bf5" (UID: "8f0068a6-2aad-417c-a347-14a7cbcc4bf5"). InnerVolumeSpecName "kube-api-access-vvfjc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 12:10:47.008962 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:10:47.008941    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f0068a6-2aad-417c-a347-14a7cbcc4bf5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8f0068a6-2aad-417c-a347-14a7cbcc4bf5" (UID: "8f0068a6-2aad-417c-a347-14a7cbcc4bf5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 12:10:47.009060 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:10:47.008946    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f0068a6-2aad-417c-a347-14a7cbcc4bf5-dshm" (OuterVolumeSpecName: "dshm") pod "8f0068a6-2aad-417c-a347-14a7cbcc4bf5" (UID: "8f0068a6-2aad-417c-a347-14a7cbcc4bf5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 12:10:47.044461 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:10:47.044412    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f0068a6-2aad-417c-a347-14a7cbcc4bf5-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8f0068a6-2aad-417c-a347-14a7cbcc4bf5" (UID: "8f0068a6-2aad-417c-a347-14a7cbcc4bf5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 12:10:47.108240 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:10:47.108115    2572 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f0068a6-2aad-417c-a347-14a7cbcc4bf5-dshm\") on node \"ip-10-0-136-194.ec2.internal\" DevicePath \"\""
May 26 12:10:47.108240 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:10:47.108142    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vvfjc\" (UniqueName: \"kubernetes.io/projected/8f0068a6-2aad-417c-a347-14a7cbcc4bf5-kube-api-access-vvfjc\") on node \"ip-10-0-136-194.ec2.internal\" DevicePath \"\""
May 26 12:10:47.108240 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:10:47.108153    2572 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8f0068a6-2aad-417c-a347-14a7cbcc4bf5-kserve-provision-location\") on node \"ip-10-0-136-194.ec2.internal\" DevicePath \"\""
May 26 12:10:47.108240 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:10:47.108162    2572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f0068a6-2aad-417c-a347-14a7cbcc4bf5-tls-certs\") on node \"ip-10-0-136-194.ec2.internal\" DevicePath \"\""
May 26 12:10:47.676823 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:10:47.676791    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9345be29-4f25-41c5-8489-c02aa4b7cea6" path="/var/lib/kubelet/pods/9345be29-4f25-41c5-8489-c02aa4b7cea6/volumes"
May 26 12:10:47.680550 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:10:47.680528    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_custom-route-timeout-pd-test-kserve-prefill-8578d996bc-4f4n5_8f0068a6-2aad-417c-a347-14a7cbcc4bf5/storage-initializer/0.log"
May 26 12:10:47.680702 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:10:47.680638    2572 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-8578d996bc-4f4n5"
May 26 12:10:47.680702 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:10:47.680648    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-8578d996bc-4f4n5" event={"ID":"8f0068a6-2aad-417c-a347-14a7cbcc4bf5","Type":"ContainerDied","Data":"139c7b87f888c240506f06c1e95a09e35aa9335602f85aaa2dff2d7f93205e6b"}
May 26 12:10:47.680702 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:10:47.680686    2572 scope.go:117] "RemoveContainer" containerID="8cc6b6b19864d9c1319126c6ad3a317501d1307610deac6bbe4de45eed4b5ace"
May 26 12:10:47.721762 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:10:47.721728    2572 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-8578d996bc-4f4n5"]
May 26 12:10:47.725917 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:10:47.725891    2572 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-8578d996bc-4f4n5"]
May 26 12:10:49.652799 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:10:49.652752    2572 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5f7f8c5848-99qts"
May 26 12:10:49.652799 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:10:49.652806    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5f7f8c5848-99qts"
May 26 12:10:49.654554 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:10:49.654521    2572 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5f7f8c5848-99qts" podUID="9b11682a-156c-4b51-a884-c0be4d37e2ec" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8001/health\": dial tcp 10.132.0.52:8001: connect: connection refused"
May 26 12:10:49.677435 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:10:49.677404    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8f0068a6-2aad-417c-a347-14a7cbcc4bf5" path="/var/lib/kubelet/pods/8f0068a6-2aad-417c-a347-14a7cbcc4bf5/volumes"
May 26 12:10:49.677754 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:10:49.677737    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5f7f8c5848-99qts"
May 26 12:10:50.697292 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:10:50.697259    2572 generic.go:358] "Generic (PLEG): container finished" podID="119a5f43-162c-4430-9b97-ce0cd2c6be9f" containerID="daad443a8d340449e2dd4d8d794ef079e69fe6e0a2ade5d325953a0b50132a56" exitCode=0
May 26 12:10:50.697292 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:10:50.697296    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7d45d77775-v9cpf" event={"ID":"119a5f43-162c-4430-9b97-ce0cd2c6be9f","Type":"ContainerDied","Data":"daad443a8d340449e2dd4d8d794ef079e69fe6e0a2ade5d325953a0b50132a56"}
May 26 12:10:51.703310 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:10:51.703267    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7d45d77775-v9cpf" event={"ID":"119a5f43-162c-4430-9b97-ce0cd2c6be9f","Type":"ContainerStarted","Data":"614a91f6b6771667ffc45213dad7a31b9f39a3aab816f93eacf3a23cd574a6ad"}
May 26 12:10:51.724401 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:10:51.724347    2572 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7d45d77775-v9cpf" podStartSLOduration=12.724329404 podStartE2EDuration="12.724329404s" podCreationTimestamp="2026-05-26 12:10:39 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 12:10:51.723940046 +0000 UTC m=+2774.626150378" watchObservedRunningTime="2026-05-26 12:10:51.724329404 +0000 UTC m=+2774.626539734"
May 26 12:10:59.652921 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:10:59.652858    2572 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5f7f8c5848-99qts" podUID="9b11682a-156c-4b51-a884-c0be4d37e2ec" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8001/health\": dial tcp 10.132.0.52:8001: connect: connection refused"
May 26 12:10:59.666959 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:10:59.666913    2572 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7d45d77775-v9cpf"
May 26 12:10:59.667148 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:10:59.666972    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7d45d77775-v9cpf"
May 26 12:10:59.668449 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:10:59.668412    2572 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7d45d77775-v9cpf" podUID="119a5f43-162c-4430-9b97-ce0cd2c6be9f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": dial tcp 10.132.0.53:8000: connect: connection refused"
May 26 12:11:09.653551 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:11:09.653500    2572 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5f7f8c5848-99qts" podUID="9b11682a-156c-4b51-a884-c0be4d37e2ec" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8001/health\": dial tcp 10.132.0.52:8001: connect: connection refused"
May 26 12:11:09.666728 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:11:09.666684    2572 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7d45d77775-v9cpf" podUID="119a5f43-162c-4430-9b97-ce0cd2c6be9f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": dial tcp 10.132.0.53:8000: connect: connection refused"
May 26 12:11:10.483960 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:11:10.483921    2572 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-788b56f4c82qtm2"]
May 26 12:11:10.484319 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:11:10.484274    2572 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-788b56f4c82qtm2" podUID="179716fc-4dad-4baf-b4f1-339ab2bcabf1" containerName="main" containerID="cri-o://f2b4359bd08724847e0482da1ee4ce73a6e1f34766293881ce5ec2e1f4caaf3a" gracePeriod=30
May 26 12:11:18.398545 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:11:18.398504    2572 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"]
May 26 12:11:18.399227 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:11:18.399167    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9345be29-4f25-41c5-8489-c02aa4b7cea6" containerName="llm-d-routing-sidecar"
May 26 12:11:18.399286 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:11:18.399235    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="9345be29-4f25-41c5-8489-c02aa4b7cea6" containerName="llm-d-routing-sidecar"
May 26 12:11:18.399286 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:11:18.399254    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f0068a6-2aad-417c-a347-14a7cbcc4bf5" containerName="storage-initializer"
May 26 12:11:18.399286 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:11:18.399265    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f0068a6-2aad-417c-a347-14a7cbcc4bf5" containerName="storage-initializer"
May 26 12:11:18.399385 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:11:18.399285    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9345be29-4f25-41c5-8489-c02aa4b7cea6" containerName="storage-initializer"
May 26 12:11:18.399385 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:11:18.399297    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="9345be29-4f25-41c5-8489-c02aa4b7cea6" containerName="storage-initializer"
May 26 12:11:18.399385 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:11:18.399326    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9345be29-4f25-41c5-8489-c02aa4b7cea6" containerName="main"
May 26 12:11:18.399385 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:11:18.399338    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="9345be29-4f25-41c5-8489-c02aa4b7cea6" containerName="main"
May 26 12:11:18.399515 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:11:18.399460    2572 memory_manager.go:356] "RemoveStaleState removing state" podUID="9345be29-4f25-41c5-8489-c02aa4b7cea6" containerName="main"
May 26 12:11:18.399515 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:11:18.399478    2572 memory_manager.go:356] "RemoveStaleState removing state" podUID="9345be29-4f25-41c5-8489-c02aa4b7cea6" containerName="llm-d-routing-sidecar"
May 26 12:11:18.399515 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:11:18.399494    2572 memory_manager.go:356] "RemoveStaleState removing state" podUID="8f0068a6-2aad-417c-a347-14a7cbcc4bf5" containerName="storage-initializer"
May 26 12:11:18.402304 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:11:18.402278    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
May 26 12:11:18.405116 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:11:18.405070    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-fb-opt-125m-route-f312f5-cb7fb8cf-dockercfg-ffgv5\""
May 26 12:11:18.405292 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:11:18.405135    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv4e643bc258191ffc517a31cd1d0ddd27-kserve-self-signed-certs\""
May 26 12:11:18.413754 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:11:18.413725    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"]
May 26 12:11:18.435815 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:11:18.435786    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/879d966b-7d17-48aa-9646-021c25f8917c-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"879d966b-7d17-48aa-9646-021c25f8917c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
May 26 12:11:18.435987 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:11:18.435821    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/879d966b-7d17-48aa-9646-021c25f8917c-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"879d966b-7d17-48aa-9646-021c25f8917c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
May 26 12:11:18.435987 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:11:18.435947    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/879d966b-7d17-48aa-9646-021c25f8917c-home\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"879d966b-7d17-48aa-9646-021c25f8917c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
May 26 12:11:18.436100 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:11:18.436000    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/879d966b-7d17-48aa-9646-021c25f8917c-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"879d966b-7d17-48aa-9646-021c25f8917c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
May 26 12:11:18.436100 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:11:18.436070    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zc5g\" (UniqueName: \"kubernetes.io/projected/879d966b-7d17-48aa-9646-021c25f8917c-kube-api-access-9zc5g\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"879d966b-7d17-48aa-9646-021c25f8917c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
May 26 12:11:18.436216 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:11:18.436110    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/879d966b-7d17-48aa-9646-021c25f8917c-dshm\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"879d966b-7d17-48aa-9646-021c25f8917c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
May 26 12:11:18.436216 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:11:18.436132    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/879d966b-7d17-48aa-9646-021c25f8917c-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"879d966b-7d17-48aa-9646-021c25f8917c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
May 26 12:11:18.536941 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:11:18.536903    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/879d966b-7d17-48aa-9646-021c25f8917c-home\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"879d966b-7d17-48aa-9646-021c25f8917c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
May 26 12:11:18.536941 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:11:18.536946    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/879d966b-7d17-48aa-9646-021c25f8917c-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"879d966b-7d17-48aa-9646-021c25f8917c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
May 26 12:11:18.537213 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:11:18.536983    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-9zc5g\" (UniqueName: \"kubernetes.io/projected/879d966b-7d17-48aa-9646-021c25f8917c-kube-api-access-9zc5g\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"879d966b-7d17-48aa-9646-021c25f8917c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
May 26 12:11:18.537213 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:11:18.537015    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/879d966b-7d17-48aa-9646-021c25f8917c-dshm\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"879d966b-7d17-48aa-9646-021c25f8917c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
May 26 12:11:18.537213 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:11:18.537045    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/879d966b-7d17-48aa-9646-021c25f8917c-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"879d966b-7d17-48aa-9646-021c25f8917c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
May 26 12:11:18.537213 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:11:18.537106    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/879d966b-7d17-48aa-9646-021c25f8917c-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"879d966b-7d17-48aa-9646-021c25f8917c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
May 26 12:11:18.537213 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:11:18.537132    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/879d966b-7d17-48aa-9646-021c25f8917c-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"879d966b-7d17-48aa-9646-021c25f8917c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
May 26 12:11:18.537485 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:11:18.537334    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/879d966b-7d17-48aa-9646-021c25f8917c-home\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"879d966b-7d17-48aa-9646-021c25f8917c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
May 26 12:11:18.537485 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:11:18.537455    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/879d966b-7d17-48aa-9646-021c25f8917c-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"879d966b-7d17-48aa-9646-021c25f8917c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
May 26 12:11:18.537606 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:11:18.537527    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/879d966b-7d17-48aa-9646-021c25f8917c-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"879d966b-7d17-48aa-9646-021c25f8917c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
May 26 12:11:18.537665 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:11:18.537625    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/879d966b-7d17-48aa-9646-021c25f8917c-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"879d966b-7d17-48aa-9646-021c25f8917c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
May 26 12:11:18.539582 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:11:18.539550    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/879d966b-7d17-48aa-9646-021c25f8917c-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"879d966b-7d17-48aa-9646-021c25f8917c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
May 26 12:11:18.539718 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:11:18.539654    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/879d966b-7d17-48aa-9646-021c25f8917c-dshm\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"879d966b-7d17-48aa-9646-021c25f8917c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
May 26 12:11:18.544845 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:11:18.544817    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zc5g\" (UniqueName: \"kubernetes.io/projected/879d966b-7d17-48aa-9646-021c25f8917c-kube-api-access-9zc5g\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"879d966b-7d17-48aa-9646-021c25f8917c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
May 26 12:11:18.716980 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:11:18.716944    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
May 26 12:11:18.859705 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:11:18.859648    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"]
May 26 12:11:18.862517 ip-10-0-136-194 kubenswrapper[2572]: W0526 12:11:18.862486    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod879d966b_7d17_48aa_9646_021c25f8917c.slice/crio-d5a2e0242a9d2f62158d95be677da512ae766bd9bcc483adc38509ecbca134a5 WatchSource:0}: Error finding container d5a2e0242a9d2f62158d95be677da512ae766bd9bcc483adc38509ecbca134a5: Status 404 returned error can't find the container with id d5a2e0242a9d2f62158d95be677da512ae766bd9bcc483adc38509ecbca134a5
May 26 12:11:19.653901 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:11:19.653859    2572 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5f7f8c5848-99qts" podUID="9b11682a-156c-4b51-a884-c0be4d37e2ec" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8001/health\": dial tcp 10.132.0.52:8001: connect: connection refused"
May 26 12:11:19.666686 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:11:19.666635    2572 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7d45d77775-v9cpf" podUID="119a5f43-162c-4430-9b97-ce0cd2c6be9f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": dial tcp 10.132.0.53:8000: connect: connection refused"
May 26 12:11:19.814412 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:11:19.814369    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" event={"ID":"879d966b-7d17-48aa-9646-021c25f8917c","Type":"ContainerStarted","Data":"67d677aff404817d1990c34179e888299a33fa79e7bdd30cd12ab180b9d60c97"}
May 26 12:11:19.814588 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:11:19.814418    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" event={"ID":"879d966b-7d17-48aa-9646-021c25f8917c","Type":"ContainerStarted","Data":"d5a2e0242a9d2f62158d95be677da512ae766bd9bcc483adc38509ecbca134a5"}
May 26 12:11:29.653052 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:11:29.652993    2572 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5f7f8c5848-99qts" podUID="9b11682a-156c-4b51-a884-c0be4d37e2ec" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8001/health\": dial tcp 10.132.0.52:8001: connect: connection refused"
May 26 12:11:29.667090 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:11:29.667046    2572 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7d45d77775-v9cpf" podUID="119a5f43-162c-4430-9b97-ce0cd2c6be9f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": dial tcp 10.132.0.53:8000: connect: connection refused"
May 26 12:11:35.887745 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:11:35.887692    2572 generic.go:358] "Generic (PLEG): container finished" podID="879d966b-7d17-48aa-9646-021c25f8917c" containerID="67d677aff404817d1990c34179e888299a33fa79e7bdd30cd12ab180b9d60c97" exitCode=0
May 26 12:11:35.888254 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:11:35.887804    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" event={"ID":"879d966b-7d17-48aa-9646-021c25f8917c","Type":"ContainerDied","Data":"67d677aff404817d1990c34179e888299a33fa79e7bdd30cd12ab180b9d60c97"}
May 26 12:11:36.895324 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:11:36.895290    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" event={"ID":"879d966b-7d17-48aa-9646-021c25f8917c","Type":"ContainerStarted","Data":"6498a82e93fd031c789c0be83a9503a996e0301e9817a8e210fed12704d0746b"}
May 26 12:11:36.918737 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:11:36.918671    2572 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" podStartSLOduration=18.918651678 podStartE2EDuration="18.918651678s" podCreationTimestamp="2026-05-26 12:11:18 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 12:11:36.915353811 +0000 UTC m=+2819.817564140" watchObservedRunningTime="2026-05-26 12:11:36.918651678 +0000 UTC m=+2819.820862005"
May 26 12:11:38.717421 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:11:38.717389    2572 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
May 26 12:11:38.718668 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:11:38.718634    2572 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" podUID="879d966b-7d17-48aa-9646-021c25f8917c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/health\": dial tcp 10.132.0.54:8000: connect: connection refused"
May 26 12:11:39.653664 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:11:39.653619    2572 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5f7f8c5848-99qts" podUID="9b11682a-156c-4b51-a884-c0be4d37e2ec" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8001/health\": dial tcp 10.132.0.52:8001: connect: connection refused"
May 26 12:11:39.666518 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:11:39.666471    2572 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7d45d77775-v9cpf" podUID="119a5f43-162c-4430-9b97-ce0cd2c6be9f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": dial tcp 10.132.0.53:8000: connect: connection refused"
May 26 12:11:40.866554 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:11:40.866523    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-788b56f4c82qtm2_179716fc-4dad-4baf-b4f1-339ab2bcabf1/main/0.log"
May 26 12:11:40.867035 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:11:40.866994    2572 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-788b56f4c82qtm2"
May 26 12:11:40.914724 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:11:40.914685    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-788b56f4c82qtm2_179716fc-4dad-4baf-b4f1-339ab2bcabf1/main/0.log"
May 26 12:11:40.915150 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:11:40.915120    2572 generic.go:358] "Generic (PLEG): container finished" podID="179716fc-4dad-4baf-b4f1-339ab2bcabf1" containerID="f2b4359bd08724847e0482da1ee4ce73a6e1f34766293881ce5ec2e1f4caaf3a" exitCode=137
May 26 12:11:40.915289 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:11:40.915176    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-788b56f4c82qtm2" event={"ID":"179716fc-4dad-4baf-b4f1-339ab2bcabf1","Type":"ContainerDied","Data":"f2b4359bd08724847e0482da1ee4ce73a6e1f34766293881ce5ec2e1f4caaf3a"}
May 26 12:11:40.915289 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:11:40.915252    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-788b56f4c82qtm2" event={"ID":"179716fc-4dad-4baf-b4f1-339ab2bcabf1","Type":"ContainerDied","Data":"a0c4688f8ed25d7de443606f27e0595cf9ec6371a6ca5388bfa40a2506febd71"}
May 26 12:11:40.915289 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:11:40.915275    2572 scope.go:117] "RemoveContainer" containerID="f2b4359bd08724847e0482da1ee4ce73a6e1f34766293881ce5ec2e1f4caaf3a"
May 26 12:11:40.915479 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:11:40.915220    2572 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-788b56f4c82qtm2"
May 26 12:11:40.925074 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:11:40.925042    2572 scope.go:117] "RemoveContainer" containerID="08b729e46667c3c98c0fdb9ac2687f3956a88f871337a135de2e159e3ed5cb6f"
May 26 12:11:40.972537 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:11:40.972499    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/179716fc-4dad-4baf-b4f1-339ab2bcabf1-tmp-dir\") pod \"179716fc-4dad-4baf-b4f1-339ab2bcabf1\" (UID: \"179716fc-4dad-4baf-b4f1-339ab2bcabf1\") "
May 26 12:11:40.972732 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:11:40.972561    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/179716fc-4dad-4baf-b4f1-339ab2bcabf1-model-cache\") pod \"179716fc-4dad-4baf-b4f1-339ab2bcabf1\" (UID: \"179716fc-4dad-4baf-b4f1-339ab2bcabf1\") "
May 26 12:11:40.972732 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:11:40.972629    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/179716fc-4dad-4baf-b4f1-339ab2bcabf1-kserve-provision-location\") pod \"179716fc-4dad-4baf-b4f1-339ab2bcabf1\" (UID: \"179716fc-4dad-4baf-b4f1-339ab2bcabf1\") "
May 26 12:11:40.972732 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:11:40.972658    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/179716fc-4dad-4baf-b4f1-339ab2bcabf1-dshm\") pod \"179716fc-4dad-4baf-b4f1-339ab2bcabf1\" (UID: \"179716fc-4dad-4baf-b4f1-339ab2bcabf1\") "
May 26 12:11:40.972732 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:11:40.972708    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9lfxk\" (UniqueName: \"kubernetes.io/projected/179716fc-4dad-4baf-b4f1-339ab2bcabf1-kube-api-access-9lfxk\") pod \"179716fc-4dad-4baf-b4f1-339ab2bcabf1\" (UID: \"179716fc-4dad-4baf-b4f1-339ab2bcabf1\") "
May 26 12:11:40.972961 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:11:40.972767    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/179716fc-4dad-4baf-b4f1-339ab2bcabf1-home\") pod \"179716fc-4dad-4baf-b4f1-339ab2bcabf1\" (UID: \"179716fc-4dad-4baf-b4f1-339ab2bcabf1\") "
May 26 12:11:40.972961 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:11:40.972822    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/179716fc-4dad-4baf-b4f1-339ab2bcabf1-tls-certs\") pod \"179716fc-4dad-4baf-b4f1-339ab2bcabf1\" (UID: \"179716fc-4dad-4baf-b4f1-339ab2bcabf1\") "
May 26 12:11:40.972961 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:11:40.972845    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/179716fc-4dad-4baf-b4f1-339ab2bcabf1-model-cache" (OuterVolumeSpecName: "model-cache") pod "179716fc-4dad-4baf-b4f1-339ab2bcabf1" (UID: "179716fc-4dad-4baf-b4f1-339ab2bcabf1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 12:11:40.973122 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:11:40.973109    2572 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/179716fc-4dad-4baf-b4f1-339ab2bcabf1-model-cache\") on node \"ip-10-0-136-194.ec2.internal\" DevicePath \"\""
May 26 12:11:40.973976 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:11:40.973917    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/179716fc-4dad-4baf-b4f1-339ab2bcabf1-home" (OuterVolumeSpecName: "home") pod "179716fc-4dad-4baf-b4f1-339ab2bcabf1" (UID: "179716fc-4dad-4baf-b4f1-339ab2bcabf1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 12:11:40.982556 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:11:40.982295    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/179716fc-4dad-4baf-b4f1-339ab2bcabf1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "179716fc-4dad-4baf-b4f1-339ab2bcabf1" (UID: "179716fc-4dad-4baf-b4f1-339ab2bcabf1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 12:11:40.982556 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:11:40.982306    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/179716fc-4dad-4baf-b4f1-339ab2bcabf1-kube-api-access-9lfxk" (OuterVolumeSpecName: "kube-api-access-9lfxk") pod "179716fc-4dad-4baf-b4f1-339ab2bcabf1" (UID: "179716fc-4dad-4baf-b4f1-339ab2bcabf1"). InnerVolumeSpecName "kube-api-access-9lfxk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 12:11:40.993183 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:11:40.993150    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/179716fc-4dad-4baf-b4f1-339ab2bcabf1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "179716fc-4dad-4baf-b4f1-339ab2bcabf1" (UID: "179716fc-4dad-4baf-b4f1-339ab2bcabf1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 12:11:40.995473 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:11:40.995438    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/179716fc-4dad-4baf-b4f1-339ab2bcabf1-dshm" (OuterVolumeSpecName: "dshm") pod "179716fc-4dad-4baf-b4f1-339ab2bcabf1" (UID: "179716fc-4dad-4baf-b4f1-339ab2bcabf1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 12:11:41.016348 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:11:41.016216    2572 scope.go:117] "RemoveContainer" containerID="f2b4359bd08724847e0482da1ee4ce73a6e1f34766293881ce5ec2e1f4caaf3a"
May 26 12:11:41.016720 ip-10-0-136-194 kubenswrapper[2572]: E0526 12:11:41.016689    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2b4359bd08724847e0482da1ee4ce73a6e1f34766293881ce5ec2e1f4caaf3a\": container with ID starting with f2b4359bd08724847e0482da1ee4ce73a6e1f34766293881ce5ec2e1f4caaf3a not found: ID does not exist" containerID="f2b4359bd08724847e0482da1ee4ce73a6e1f34766293881ce5ec2e1f4caaf3a"
May 26 12:11:41.016801 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:11:41.016734    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2b4359bd08724847e0482da1ee4ce73a6e1f34766293881ce5ec2e1f4caaf3a"} err="failed to get container status \"f2b4359bd08724847e0482da1ee4ce73a6e1f34766293881ce5ec2e1f4caaf3a\": rpc error: code = NotFound desc = could not find container \"f2b4359bd08724847e0482da1ee4ce73a6e1f34766293881ce5ec2e1f4caaf3a\": container with ID starting with f2b4359bd08724847e0482da1ee4ce73a6e1f34766293881ce5ec2e1f4caaf3a not found: ID does not exist"
May 26 12:11:41.016801 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:11:41.016764    2572 scope.go:117] "RemoveContainer" containerID="08b729e46667c3c98c0fdb9ac2687f3956a88f871337a135de2e159e3ed5cb6f"
May 26 12:11:41.017140 ip-10-0-136-194 kubenswrapper[2572]: E0526 12:11:41.017115    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08b729e46667c3c98c0fdb9ac2687f3956a88f871337a135de2e159e3ed5cb6f\": container with ID starting with 08b729e46667c3c98c0fdb9ac2687f3956a88f871337a135de2e159e3ed5cb6f not found: ID does not exist" containerID="08b729e46667c3c98c0fdb9ac2687f3956a88f871337a135de2e159e3ed5cb6f"
May 26 12:11:41.017245 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:11:41.017150    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08b729e46667c3c98c0fdb9ac2687f3956a88f871337a135de2e159e3ed5cb6f"} err="failed to get container status \"08b729e46667c3c98c0fdb9ac2687f3956a88f871337a135de2e159e3ed5cb6f\": rpc error: code = NotFound desc = could not find container \"08b729e46667c3c98c0fdb9ac2687f3956a88f871337a135de2e159e3ed5cb6f\": container with ID starting with 08b729e46667c3c98c0fdb9ac2687f3956a88f871337a135de2e159e3ed5cb6f not found: ID does not exist"
May 26 12:11:41.068618 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:11:41.068560    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/179716fc-4dad-4baf-b4f1-339ab2bcabf1-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "179716fc-4dad-4baf-b4f1-339ab2bcabf1" (UID: "179716fc-4dad-4baf-b4f1-339ab2bcabf1"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 12:11:41.074613 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:11:41.074584    2572 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/179716fc-4dad-4baf-b4f1-339ab2bcabf1-home\") on node \"ip-10-0-136-194.ec2.internal\" DevicePath \"\""
May 26 12:11:41.074698 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:11:41.074622    2572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/179716fc-4dad-4baf-b4f1-339ab2bcabf1-tls-certs\") on node \"ip-10-0-136-194.ec2.internal\" DevicePath \"\""
May 26 12:11:41.074698 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:11:41.074640    2572 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/179716fc-4dad-4baf-b4f1-339ab2bcabf1-tmp-dir\") on node \"ip-10-0-136-194.ec2.internal\" DevicePath \"\""
May 26 12:11:41.074698 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:11:41.074655    2572 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/179716fc-4dad-4baf-b4f1-339ab2bcabf1-kserve-provision-location\") on node \"ip-10-0-136-194.ec2.internal\" DevicePath \"\""
May 26 12:11:41.074698 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:11:41.074669    2572 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/179716fc-4dad-4baf-b4f1-339ab2bcabf1-dshm\") on node \"ip-10-0-136-194.ec2.internal\" DevicePath \"\""
May 26 12:11:41.074698 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:11:41.074683    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9lfxk\" (UniqueName: \"kubernetes.io/projected/179716fc-4dad-4baf-b4f1-339ab2bcabf1-kube-api-access-9lfxk\") on node \"ip-10-0-136-194.ec2.internal\" DevicePath \"\""
May 26 12:11:41.244407 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:11:41.244370    2572 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-788b56f4c82qtm2"]
May 26 12:11:41.248355 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:11:41.248327    2572 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-788b56f4c82qtm2"]
May 26 12:11:41.677563 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:11:41.677521    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="179716fc-4dad-4baf-b4f1-339ab2bcabf1" path="/var/lib/kubelet/pods/179716fc-4dad-4baf-b4f1-339ab2bcabf1/volumes"
May 26 12:11:48.717448 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:11:48.717412    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
May 26 12:11:48.717899 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:11:48.717642    2572 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" podUID="879d966b-7d17-48aa-9646-021c25f8917c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/health\": dial tcp 10.132.0.54:8000: connect: connection refused"
May 26 12:11:49.653108 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:11:49.653063    2572 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5f7f8c5848-99qts" podUID="9b11682a-156c-4b51-a884-c0be4d37e2ec" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8001/health\": dial tcp 10.132.0.52:8001: connect: connection refused"
May 26 12:11:49.666742 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:11:49.666705    2572 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7d45d77775-v9cpf" podUID="119a5f43-162c-4430-9b97-ce0cd2c6be9f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": dial tcp 10.132.0.53:8000: connect: connection refused"
May 26 12:11:58.718115 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:11:58.718068    2572 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" podUID="879d966b-7d17-48aa-9646-021c25f8917c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/health\": dial tcp 10.132.0.54:8000: connect: connection refused"
May 26 12:11:59.653726 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:11:59.653666    2572 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5f7f8c5848-99qts" podUID="9b11682a-156c-4b51-a884-c0be4d37e2ec" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8001/health\": dial tcp 10.132.0.52:8001: connect: connection refused"
May 26 12:11:59.667009 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:11:59.666953    2572 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7d45d77775-v9cpf" podUID="119a5f43-162c-4430-9b97-ce0cd2c6be9f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": dial tcp 10.132.0.53:8000: connect: connection refused"
May 26 12:12:08.718240 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:12:08.718176    2572 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" podUID="879d966b-7d17-48aa-9646-021c25f8917c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/health\": dial tcp 10.132.0.54:8000: connect: connection refused"
May 26 12:12:09.652781 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:12:09.652731    2572 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5f7f8c5848-99qts" podUID="9b11682a-156c-4b51-a884-c0be4d37e2ec" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8001/health\": dial tcp 10.132.0.52:8001: connect: connection refused"
May 26 12:12:09.666363 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:12:09.666328    2572 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7d45d77775-v9cpf" podUID="119a5f43-162c-4430-9b97-ce0cd2c6be9f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": dial tcp 10.132.0.53:8000: connect: connection refused"
May 26 12:12:18.717460 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:12:18.717410    2572 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" podUID="879d966b-7d17-48aa-9646-021c25f8917c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/health\": dial tcp 10.132.0.54:8000: connect: connection refused"
May 26 12:12:19.653290 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:12:19.653231    2572 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5f7f8c5848-99qts" podUID="9b11682a-156c-4b51-a884-c0be4d37e2ec" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8001/health\": dial tcp 10.132.0.52:8001: connect: connection refused"
May 26 12:12:19.666319 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:12:19.666277    2572 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7d45d77775-v9cpf" podUID="119a5f43-162c-4430-9b97-ce0cd2c6be9f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": dial tcp 10.132.0.53:8000: connect: connection refused"
May 26 12:12:28.718003 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:12:28.717952    2572 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" podUID="879d966b-7d17-48aa-9646-021c25f8917c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/health\": dial tcp 10.132.0.54:8000: connect: connection refused"
May 26 12:12:29.653492 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:12:29.653445    2572 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5f7f8c5848-99qts" podUID="9b11682a-156c-4b51-a884-c0be4d37e2ec" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8001/health\": dial tcp 10.132.0.52:8001: connect: connection refused"
May 26 12:12:29.667140 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:12:29.667095    2572 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7d45d77775-v9cpf" podUID="119a5f43-162c-4430-9b97-ce0cd2c6be9f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": dial tcp 10.132.0.53:8000: connect: connection refused"
May 26 12:12:38.717748 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:12:38.717702    2572 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" podUID="879d966b-7d17-48aa-9646-021c25f8917c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/health\": dial tcp 10.132.0.54:8000: connect: connection refused"
May 26 12:12:39.653163 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:12:39.653112    2572 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5f7f8c5848-99qts" podUID="9b11682a-156c-4b51-a884-c0be4d37e2ec" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8001/health\": dial tcp 10.132.0.52:8001: connect: connection refused"
May 26 12:12:39.666530 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:12:39.666485    2572 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7d45d77775-v9cpf" podUID="119a5f43-162c-4430-9b97-ce0cd2c6be9f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": dial tcp 10.132.0.53:8000: connect: connection refused"
May 26 12:12:48.717851 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:12:48.717811    2572 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" podUID="879d966b-7d17-48aa-9646-021c25f8917c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/health\": dial tcp 10.132.0.54:8000: connect: connection refused"
May 26 12:12:49.666512 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:12:49.666468    2572 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7d45d77775-v9cpf" podUID="119a5f43-162c-4430-9b97-ce0cd2c6be9f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": dial tcp 10.132.0.53:8000: connect: connection refused"
May 26 12:12:49.668454 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:12:49.668431    2572 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5f7f8c5848-99qts"
May 26 12:12:49.683867 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:12:49.683839    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5f7f8c5848-99qts"
May 26 12:12:58.717847 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:12:58.717806    2572 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" podUID="879d966b-7d17-48aa-9646-021c25f8917c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/health\": dial tcp 10.132.0.54:8000: connect: connection refused"
May 26 12:12:59.666926 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:12:59.666888    2572 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7d45d77775-v9cpf" podUID="119a5f43-162c-4430-9b97-ce0cd2c6be9f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": dial tcp 10.132.0.53:8000: connect: connection refused"
May 26 12:13:08.717494 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:13:08.717450    2572 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" podUID="879d966b-7d17-48aa-9646-021c25f8917c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/health\": dial tcp 10.132.0.54:8000: connect: connection refused"
May 26 12:13:09.667180 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:13:09.667142    2572 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7d45d77775-v9cpf" podUID="119a5f43-162c-4430-9b97-ce0cd2c6be9f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": dial tcp 10.132.0.53:8000: connect: connection refused"
May 26 12:13:18.717947 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:13:18.717907    2572 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" podUID="879d966b-7d17-48aa-9646-021c25f8917c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/health\": dial tcp 10.132.0.54:8000: connect: connection refused"
May 26 12:13:19.666732 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:13:19.666688    2572 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7d45d77775-v9cpf" podUID="119a5f43-162c-4430-9b97-ce0cd2c6be9f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": dial tcp 10.132.0.53:8000: connect: connection refused"
May 26 12:13:28.718019 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:13:28.717973    2572 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" podUID="879d966b-7d17-48aa-9646-021c25f8917c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/health\": dial tcp 10.132.0.54:8000: connect: connection refused"
May 26 12:13:29.666402 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:13:29.666364    2572 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7d45d77775-v9cpf" podUID="119a5f43-162c-4430-9b97-ce0cd2c6be9f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": dial tcp 10.132.0.53:8000: connect: connection refused"
May 26 12:13:38.717876 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:13:38.717833    2572 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" podUID="879d966b-7d17-48aa-9646-021c25f8917c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/health\": dial tcp 10.132.0.54:8000: connect: connection refused"
May 26 12:13:39.667376 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:13:39.667332    2572 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7d45d77775-v9cpf" podUID="119a5f43-162c-4430-9b97-ce0cd2c6be9f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": dial tcp 10.132.0.53:8000: connect: connection refused"
May 26 12:13:48.727175 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:13:48.727143    2572 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
May 26 12:13:48.734616 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:13:48.734591    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
May 26 12:13:49.681121 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:13:49.681085    2572 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7d45d77775-v9cpf"
May 26 12:13:49.689562 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:13:49.689535    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7d45d77775-v9cpf"
May 26 12:13:59.176434 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:13:59.176395    2572 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"]
May 26 12:13:59.176902 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:13:59.176799    2572 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" podUID="879d966b-7d17-48aa-9646-021c25f8917c" containerName="main" containerID="cri-o://6498a82e93fd031c789c0be83a9503a996e0301e9817a8e210fed12704d0746b" gracePeriod=30
May 26 12:14:00.231162 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:00.231138    2572 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
May 26 12:14:00.320366 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:00.320275    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/879d966b-7d17-48aa-9646-021c25f8917c-home\") pod \"879d966b-7d17-48aa-9646-021c25f8917c\" (UID: \"879d966b-7d17-48aa-9646-021c25f8917c\") "
May 26 12:14:00.320366 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:00.320315    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/879d966b-7d17-48aa-9646-021c25f8917c-kserve-provision-location\") pod \"879d966b-7d17-48aa-9646-021c25f8917c\" (UID: \"879d966b-7d17-48aa-9646-021c25f8917c\") "
May 26 12:14:00.320366 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:00.320366    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/879d966b-7d17-48aa-9646-021c25f8917c-model-cache\") pod \"879d966b-7d17-48aa-9646-021c25f8917c\" (UID: \"879d966b-7d17-48aa-9646-021c25f8917c\") "
May 26 12:14:00.320603 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:00.320381    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/879d966b-7d17-48aa-9646-021c25f8917c-dshm\") pod \"879d966b-7d17-48aa-9646-021c25f8917c\" (UID: \"879d966b-7d17-48aa-9646-021c25f8917c\") "
May 26 12:14:00.320603 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:00.320415    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/879d966b-7d17-48aa-9646-021c25f8917c-tls-certs\") pod \"879d966b-7d17-48aa-9646-021c25f8917c\" (UID: \"879d966b-7d17-48aa-9646-021c25f8917c\") "
May 26 12:14:00.320603 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:00.320454    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/879d966b-7d17-48aa-9646-021c25f8917c-tmp-dir\") pod \"879d966b-7d17-48aa-9646-021c25f8917c\" (UID: \"879d966b-7d17-48aa-9646-021c25f8917c\") "
May 26 12:14:00.320603 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:00.320480    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9zc5g\" (UniqueName: \"kubernetes.io/projected/879d966b-7d17-48aa-9646-021c25f8917c-kube-api-access-9zc5g\") pod \"879d966b-7d17-48aa-9646-021c25f8917c\" (UID: \"879d966b-7d17-48aa-9646-021c25f8917c\") "
May 26 12:14:00.320769 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:00.320636    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/879d966b-7d17-48aa-9646-021c25f8917c-model-cache" (OuterVolumeSpecName: "model-cache") pod "879d966b-7d17-48aa-9646-021c25f8917c" (UID: "879d966b-7d17-48aa-9646-021c25f8917c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 12:14:00.320827 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:00.320811    2572 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/879d966b-7d17-48aa-9646-021c25f8917c-model-cache\") on node \"ip-10-0-136-194.ec2.internal\" DevicePath \"\""
May 26 12:14:00.321010 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:00.320936    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/879d966b-7d17-48aa-9646-021c25f8917c-home" (OuterVolumeSpecName: "home") pod "879d966b-7d17-48aa-9646-021c25f8917c" (UID: "879d966b-7d17-48aa-9646-021c25f8917c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 12:14:00.322932 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:00.322905    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/879d966b-7d17-48aa-9646-021c25f8917c-dshm" (OuterVolumeSpecName: "dshm") pod "879d966b-7d17-48aa-9646-021c25f8917c" (UID: "879d966b-7d17-48aa-9646-021c25f8917c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 12:14:00.323044 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:00.322930    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/879d966b-7d17-48aa-9646-021c25f8917c-kube-api-access-9zc5g" (OuterVolumeSpecName: "kube-api-access-9zc5g") pod "879d966b-7d17-48aa-9646-021c25f8917c" (UID: "879d966b-7d17-48aa-9646-021c25f8917c"). InnerVolumeSpecName "kube-api-access-9zc5g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 12:14:00.323468 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:00.323440    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/879d966b-7d17-48aa-9646-021c25f8917c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "879d966b-7d17-48aa-9646-021c25f8917c" (UID: "879d966b-7d17-48aa-9646-021c25f8917c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 12:14:00.337671 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:00.337641    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/879d966b-7d17-48aa-9646-021c25f8917c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "879d966b-7d17-48aa-9646-021c25f8917c" (UID: "879d966b-7d17-48aa-9646-021c25f8917c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 12:14:00.374109 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:00.374056    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/879d966b-7d17-48aa-9646-021c25f8917c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "879d966b-7d17-48aa-9646-021c25f8917c" (UID: "879d966b-7d17-48aa-9646-021c25f8917c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 12:14:00.421299 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:00.421268    2572 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/879d966b-7d17-48aa-9646-021c25f8917c-home\") on node \"ip-10-0-136-194.ec2.internal\" DevicePath \"\""
May 26 12:14:00.421299 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:00.421296    2572 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/879d966b-7d17-48aa-9646-021c25f8917c-kserve-provision-location\") on node \"ip-10-0-136-194.ec2.internal\" DevicePath \"\""
May 26 12:14:00.421299 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:00.421306    2572 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/879d966b-7d17-48aa-9646-021c25f8917c-dshm\") on node \"ip-10-0-136-194.ec2.internal\" DevicePath \"\""
May 26 12:14:00.421535 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:00.421315    2572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/879d966b-7d17-48aa-9646-021c25f8917c-tls-certs\") on node \"ip-10-0-136-194.ec2.internal\" DevicePath \"\""
May 26 12:14:00.421535 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:00.421325    2572 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/879d966b-7d17-48aa-9646-021c25f8917c-tmp-dir\") on node \"ip-10-0-136-194.ec2.internal\" DevicePath \"\""
May 26 12:14:00.421535 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:00.421334    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9zc5g\" (UniqueName: \"kubernetes.io/projected/879d966b-7d17-48aa-9646-021c25f8917c-kube-api-access-9zc5g\") on node \"ip-10-0-136-194.ec2.internal\" DevicePath \"\""
May 26 12:14:00.475690 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:00.475648    2572 generic.go:358] "Generic (PLEG): container finished" podID="879d966b-7d17-48aa-9646-021c25f8917c" containerID="6498a82e93fd031c789c0be83a9503a996e0301e9817a8e210fed12704d0746b" exitCode=0
May 26 12:14:00.475901 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:00.475732    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" event={"ID":"879d966b-7d17-48aa-9646-021c25f8917c","Type":"ContainerDied","Data":"6498a82e93fd031c789c0be83a9503a996e0301e9817a8e210fed12704d0746b"}
May 26 12:14:00.475901 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:00.475750    2572 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
May 26 12:14:00.475901 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:00.475780    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" event={"ID":"879d966b-7d17-48aa-9646-021c25f8917c","Type":"ContainerDied","Data":"d5a2e0242a9d2f62158d95be677da512ae766bd9bcc483adc38509ecbca134a5"}
May 26 12:14:00.475901 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:00.475800    2572 scope.go:117] "RemoveContainer" containerID="6498a82e93fd031c789c0be83a9503a996e0301e9817a8e210fed12704d0746b"
May 26 12:14:00.486017 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:00.486000    2572 scope.go:117] "RemoveContainer" containerID="67d677aff404817d1990c34179e888299a33fa79e7bdd30cd12ab180b9d60c97"
May 26 12:14:00.499359 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:00.499336    2572 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"]
May 26 12:14:00.500452 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:00.500431    2572 scope.go:117] "RemoveContainer" containerID="6498a82e93fd031c789c0be83a9503a996e0301e9817a8e210fed12704d0746b"
May 26 12:14:00.500776 ip-10-0-136-194 kubenswrapper[2572]: E0526 12:14:00.500750    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6498a82e93fd031c789c0be83a9503a996e0301e9817a8e210fed12704d0746b\": container with ID starting with 6498a82e93fd031c789c0be83a9503a996e0301e9817a8e210fed12704d0746b not found: ID does not exist" containerID="6498a82e93fd031c789c0be83a9503a996e0301e9817a8e210fed12704d0746b"
May 26 12:14:00.501049 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:00.500784    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6498a82e93fd031c789c0be83a9503a996e0301e9817a8e210fed12704d0746b"} err="failed to get container status \"6498a82e93fd031c789c0be83a9503a996e0301e9817a8e210fed12704d0746b\": rpc error: code = NotFound desc = could not find container \"6498a82e93fd031c789c0be83a9503a996e0301e9817a8e210fed12704d0746b\": container with ID starting with 6498a82e93fd031c789c0be83a9503a996e0301e9817a8e210fed12704d0746b not found: ID does not exist"
May 26 12:14:00.501049 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:00.500803    2572 scope.go:117] "RemoveContainer" containerID="67d677aff404817d1990c34179e888299a33fa79e7bdd30cd12ab180b9d60c97"
May 26 12:14:00.501213 ip-10-0-136-194 kubenswrapper[2572]: E0526 12:14:00.501120    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67d677aff404817d1990c34179e888299a33fa79e7bdd30cd12ab180b9d60c97\": container with ID starting with 67d677aff404817d1990c34179e888299a33fa79e7bdd30cd12ab180b9d60c97 not found: ID does not exist" containerID="67d677aff404817d1990c34179e888299a33fa79e7bdd30cd12ab180b9d60c97"
May 26 12:14:00.501213 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:00.501150    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67d677aff404817d1990c34179e888299a33fa79e7bdd30cd12ab180b9d60c97"} err="failed to get container status \"67d677aff404817d1990c34179e888299a33fa79e7bdd30cd12ab180b9d60c97\": rpc error: code = NotFound desc = could not find container \"67d677aff404817d1990c34179e888299a33fa79e7bdd30cd12ab180b9d60c97\": container with ID starting with 67d677aff404817d1990c34179e888299a33fa79e7bdd30cd12ab180b9d60c97 not found: ID does not exist"
May 26 12:14:00.504054 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:00.504028    2572 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"]
May 26 12:14:01.679966 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:01.679904    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="879d966b-7d17-48aa-9646-021c25f8917c" path="/var/lib/kubelet/pods/879d966b-7d17-48aa-9646-021c25f8917c/volumes"
May 26 12:14:04.913490 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:04.913452    2572 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5f7f8c5848-99qts"]
May 26 12:14:04.913965 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:04.913920    2572 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5f7f8c5848-99qts" podUID="9b11682a-156c-4b51-a884-c0be4d37e2ec" containerName="main" containerID="cri-o://70138a75f3c3d1a2cc0a1cc5bdb1a80d705880a5acbd87e5975f0c6d6231cfe3" gracePeriod=30
May 26 12:14:04.921485 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:04.921463    2572 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7d45d77775-v9cpf"]
May 26 12:14:04.921719 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:04.921695    2572 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7d45d77775-v9cpf" podUID="119a5f43-162c-4430-9b97-ce0cd2c6be9f" containerName="main" containerID="cri-o://614a91f6b6771667ffc45213dad7a31b9f39a3aab816f93eacf3a23cd574a6ad" gracePeriod=30
May 26 12:14:22.492578 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:22.492545    2572 ???:1] "http: TLS handshake error from 10.0.133.125:35114: EOF"
May 26 12:14:22.503590 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:22.503564    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-5f7f8c5848-99qts_9b11682a-156c-4b51-a884-c0be4d37e2ec/main/0.log"
May 26 12:14:22.521989 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:22.521962    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-5f7f8c5848-99qts_9b11682a-156c-4b51-a884-c0be4d37e2ec/llm-d-routing-sidecar/0.log"
May 26 12:14:22.533554 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:22.533524    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-5f7f8c5848-99qts_9b11682a-156c-4b51-a884-c0be4d37e2ec/storage-initializer/0.log"
May 26 12:14:22.554697 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:22.554678    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-prefill-7d45d77775-v9cpf_119a5f43-162c-4430-9b97-ce0cd2c6be9f/main/0.log"
May 26 12:14:22.566023 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:22.566001    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-prefill-7d45d77775-v9cpf_119a5f43-162c-4430-9b97-ce0cd2c6be9f/storage-initializer/0.log"
May 26 12:14:23.622535 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:23.622502    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-5f7f8c5848-99qts_9b11682a-156c-4b51-a884-c0be4d37e2ec/main/0.log"
May 26 12:14:23.630329 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:23.630302    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-5f7f8c5848-99qts_9b11682a-156c-4b51-a884-c0be4d37e2ec/llm-d-routing-sidecar/0.log"
May 26 12:14:23.644153 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:23.644118    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-5f7f8c5848-99qts_9b11682a-156c-4b51-a884-c0be4d37e2ec/storage-initializer/0.log"
May 26 12:14:23.665219 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:23.665180    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-prefill-7d45d77775-v9cpf_119a5f43-162c-4430-9b97-ce0cd2c6be9f/main/0.log"
May 26 12:14:23.672319 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:23.672297    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-prefill-7d45d77775-v9cpf_119a5f43-162c-4430-9b97-ce0cd2c6be9f/storage-initializer/0.log"
May 26 12:14:24.696286 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:24.696252    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-5f7f8c5848-99qts_9b11682a-156c-4b51-a884-c0be4d37e2ec/main/0.log"
May 26 12:14:24.704773 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:24.704751    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-5f7f8c5848-99qts_9b11682a-156c-4b51-a884-c0be4d37e2ec/llm-d-routing-sidecar/0.log"
May 26 12:14:24.714579 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:24.714556    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-5f7f8c5848-99qts_9b11682a-156c-4b51-a884-c0be4d37e2ec/storage-initializer/0.log"
May 26 12:14:24.733857 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:24.733842    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-prefill-7d45d77775-v9cpf_119a5f43-162c-4430-9b97-ce0cd2c6be9f/main/0.log"
May 26 12:14:24.740734 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:24.740716    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-prefill-7d45d77775-v9cpf_119a5f43-162c-4430-9b97-ce0cd2c6be9f/storage-initializer/0.log"
May 26 12:14:25.748294 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:25.748266    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-5f7f8c5848-99qts_9b11682a-156c-4b51-a884-c0be4d37e2ec/main/0.log"
May 26 12:14:25.755608 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:25.755588    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-5f7f8c5848-99qts_9b11682a-156c-4b51-a884-c0be4d37e2ec/llm-d-routing-sidecar/0.log"
May 26 12:14:25.765964 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:25.765944    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-5f7f8c5848-99qts_9b11682a-156c-4b51-a884-c0be4d37e2ec/storage-initializer/0.log"
May 26 12:14:25.785560 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:25.785532    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-prefill-7d45d77775-v9cpf_119a5f43-162c-4430-9b97-ce0cd2c6be9f/main/0.log"
May 26 12:14:25.799146 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:25.799125    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-prefill-7d45d77775-v9cpf_119a5f43-162c-4430-9b97-ce0cd2c6be9f/storage-initializer/0.log"
May 26 12:14:26.802203 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:26.802145    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-5f7f8c5848-99qts_9b11682a-156c-4b51-a884-c0be4d37e2ec/main/0.log"
May 26 12:14:26.810791 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:26.810769    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-5f7f8c5848-99qts_9b11682a-156c-4b51-a884-c0be4d37e2ec/llm-d-routing-sidecar/0.log"
May 26 12:14:26.825272 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:26.825252    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-5f7f8c5848-99qts_9b11682a-156c-4b51-a884-c0be4d37e2ec/storage-initializer/0.log"
May 26 12:14:26.849025 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:26.849006    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-prefill-7d45d77775-v9cpf_119a5f43-162c-4430-9b97-ce0cd2c6be9f/main/0.log"
May 26 12:14:26.862600 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:26.862578    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-prefill-7d45d77775-v9cpf_119a5f43-162c-4430-9b97-ce0cd2c6be9f/storage-initializer/0.log"
May 26 12:14:27.919512 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:27.919478    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-5f7f8c5848-99qts_9b11682a-156c-4b51-a884-c0be4d37e2ec/main/0.log"
May 26 12:14:27.929528 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:27.929504    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-5f7f8c5848-99qts_9b11682a-156c-4b51-a884-c0be4d37e2ec/llm-d-routing-sidecar/0.log"
May 26 12:14:27.945334 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:27.945310    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-5f7f8c5848-99qts_9b11682a-156c-4b51-a884-c0be4d37e2ec/storage-initializer/0.log"
May 26 12:14:27.967132 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:27.967113    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-prefill-7d45d77775-v9cpf_119a5f43-162c-4430-9b97-ce0cd2c6be9f/main/0.log"
May 26 12:14:27.989692 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:27.989668    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-prefill-7d45d77775-v9cpf_119a5f43-162c-4430-9b97-ce0cd2c6be9f/storage-initializer/0.log"
May 26 12:14:29.042576 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:29.042545    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-5f7f8c5848-99qts_9b11682a-156c-4b51-a884-c0be4d37e2ec/main/0.log"
May 26 12:14:29.050510 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:29.050481    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-5f7f8c5848-99qts_9b11682a-156c-4b51-a884-c0be4d37e2ec/llm-d-routing-sidecar/0.log"
May 26 12:14:29.061244 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:29.061223    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-5f7f8c5848-99qts_9b11682a-156c-4b51-a884-c0be4d37e2ec/storage-initializer/0.log"
May 26 12:14:29.080771 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:29.080748    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-prefill-7d45d77775-v9cpf_119a5f43-162c-4430-9b97-ce0cd2c6be9f/main/0.log"
May 26 12:14:29.088619 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:29.088596    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-prefill-7d45d77775-v9cpf_119a5f43-162c-4430-9b97-ce0cd2c6be9f/storage-initializer/0.log"
May 26 12:14:30.111523 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:30.111492    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-5f7f8c5848-99qts_9b11682a-156c-4b51-a884-c0be4d37e2ec/main/0.log"
May 26 12:14:30.119437 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:30.119417    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-5f7f8c5848-99qts_9b11682a-156c-4b51-a884-c0be4d37e2ec/llm-d-routing-sidecar/0.log"
May 26 12:14:30.134988 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:30.134966    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-5f7f8c5848-99qts_9b11682a-156c-4b51-a884-c0be4d37e2ec/storage-initializer/0.log"
May 26 12:14:30.155834 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:30.155810    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-prefill-7d45d77775-v9cpf_119a5f43-162c-4430-9b97-ce0cd2c6be9f/main/0.log"
May 26 12:14:30.165828 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:30.165802    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-prefill-7d45d77775-v9cpf_119a5f43-162c-4430-9b97-ce0cd2c6be9f/storage-initializer/0.log"
May 26 12:14:31.247738 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:31.247710    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-5f7f8c5848-99qts_9b11682a-156c-4b51-a884-c0be4d37e2ec/main/0.log"
May 26 12:14:31.263100 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:31.263068    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-5f7f8c5848-99qts_9b11682a-156c-4b51-a884-c0be4d37e2ec/llm-d-routing-sidecar/0.log"
May 26 12:14:31.288219 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:31.288183    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-5f7f8c5848-99qts_9b11682a-156c-4b51-a884-c0be4d37e2ec/storage-initializer/0.log"
May 26 12:14:31.315620 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:31.315601    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-prefill-7d45d77775-v9cpf_119a5f43-162c-4430-9b97-ce0cd2c6be9f/main/0.log"
May 26 12:14:31.325484 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:31.325465    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-prefill-7d45d77775-v9cpf_119a5f43-162c-4430-9b97-ce0cd2c6be9f/storage-initializer/0.log"
May 26 12:14:32.421544 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:32.421510    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-5f7f8c5848-99qts_9b11682a-156c-4b51-a884-c0be4d37e2ec/main/0.log"
May 26 12:14:32.430627 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:32.430602    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-5f7f8c5848-99qts_9b11682a-156c-4b51-a884-c0be4d37e2ec/llm-d-routing-sidecar/0.log"
May 26 12:14:32.446205 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:32.446163    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-5f7f8c5848-99qts_9b11682a-156c-4b51-a884-c0be4d37e2ec/storage-initializer/0.log"
May 26 12:14:32.473561 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:32.473534    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-prefill-7d45d77775-v9cpf_119a5f43-162c-4430-9b97-ce0cd2c6be9f/main/0.log"
May 26 12:14:32.493308 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:32.493285    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-prefill-7d45d77775-v9cpf_119a5f43-162c-4430-9b97-ce0cd2c6be9f/storage-initializer/0.log"
May 26 12:14:33.900714 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:33.900686    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-5f7f8c5848-99qts_9b11682a-156c-4b51-a884-c0be4d37e2ec/main/0.log"
May 26 12:14:33.917373 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:33.917339    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-5f7f8c5848-99qts_9b11682a-156c-4b51-a884-c0be4d37e2ec/llm-d-routing-sidecar/0.log"
May 26 12:14:33.931295 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:33.931268    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-5f7f8c5848-99qts_9b11682a-156c-4b51-a884-c0be4d37e2ec/storage-initializer/0.log"
May 26 12:14:33.968690 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:33.968662    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-prefill-7d45d77775-v9cpf_119a5f43-162c-4430-9b97-ce0cd2c6be9f/main/0.log"
May 26 12:14:34.002047 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:34.002016    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-prefill-7d45d77775-v9cpf_119a5f43-162c-4430-9b97-ce0cd2c6be9f/storage-initializer/0.log"
May 26 12:14:34.914280 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:34.914240    2572 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5f7f8c5848-99qts" podUID="9b11682a-156c-4b51-a884-c0be4d37e2ec" containerName="llm-d-routing-sidecar" containerID="cri-o://7daef1c0674dbcd8bff1d5c55f6fce7d309cf4243214b77537f86599a2c6ba5c" gracePeriod=2
May 26 12:14:35.036168 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:35.036120    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-5f7f8c5848-99qts_9b11682a-156c-4b51-a884-c0be4d37e2ec/main/0.log"
May 26 12:14:35.044319 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:35.044295    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-5f7f8c5848-99qts_9b11682a-156c-4b51-a884-c0be4d37e2ec/llm-d-routing-sidecar/0.log"
May 26 12:14:35.054986 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:35.054965    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-5f7f8c5848-99qts_9b11682a-156c-4b51-a884-c0be4d37e2ec/storage-initializer/0.log"
May 26 12:14:35.077223 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:35.077177    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-prefill-7d45d77775-v9cpf_119a5f43-162c-4430-9b97-ce0cd2c6be9f/main/0.log"
May 26 12:14:35.094273 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:35.094249    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-prefill-7d45d77775-v9cpf_119a5f43-162c-4430-9b97-ce0cd2c6be9f/storage-initializer/0.log"
May 26 12:14:35.226402 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:35.226383    2572 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7d45d77775-v9cpf"
May 26 12:14:35.235815 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:35.235794    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/119a5f43-162c-4430-9b97-ce0cd2c6be9f-model-cache\") pod \"119a5f43-162c-4430-9b97-ce0cd2c6be9f\" (UID: \"119a5f43-162c-4430-9b97-ce0cd2c6be9f\") "
May 26 12:14:35.235917 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:35.235825    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bmztk\" (UniqueName: \"kubernetes.io/projected/119a5f43-162c-4430-9b97-ce0cd2c6be9f-kube-api-access-bmztk\") pod \"119a5f43-162c-4430-9b97-ce0cd2c6be9f\" (UID: \"119a5f43-162c-4430-9b97-ce0cd2c6be9f\") "
May 26 12:14:35.235917 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:35.235882    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/119a5f43-162c-4430-9b97-ce0cd2c6be9f-tmp-dir\") pod \"119a5f43-162c-4430-9b97-ce0cd2c6be9f\" (UID: \"119a5f43-162c-4430-9b97-ce0cd2c6be9f\") "
May 26 12:14:35.235917 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:35.235910    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/119a5f43-162c-4430-9b97-ce0cd2c6be9f-tls-certs\") pod \"119a5f43-162c-4430-9b97-ce0cd2c6be9f\" (UID: \"119a5f43-162c-4430-9b97-ce0cd2c6be9f\") "
May 26 12:14:35.236054 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:35.235963    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/119a5f43-162c-4430-9b97-ce0cd2c6be9f-dshm\") pod \"119a5f43-162c-4430-9b97-ce0cd2c6be9f\" (UID: \"119a5f43-162c-4430-9b97-ce0cd2c6be9f\") "
May 26 12:14:35.236054 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:35.236002    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/119a5f43-162c-4430-9b97-ce0cd2c6be9f-home\") pod \"119a5f43-162c-4430-9b97-ce0cd2c6be9f\" (UID: \"119a5f43-162c-4430-9b97-ce0cd2c6be9f\") "
May 26 12:14:35.236054 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:35.236045    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/119a5f43-162c-4430-9b97-ce0cd2c6be9f-kserve-provision-location\") pod \"119a5f43-162c-4430-9b97-ce0cd2c6be9f\" (UID: \"119a5f43-162c-4430-9b97-ce0cd2c6be9f\") "
May 26 12:14:35.236225 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:35.236142    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/119a5f43-162c-4430-9b97-ce0cd2c6be9f-model-cache" (OuterVolumeSpecName: "model-cache") pod "119a5f43-162c-4430-9b97-ce0cd2c6be9f" (UID: "119a5f43-162c-4430-9b97-ce0cd2c6be9f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 12:14:35.236388 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:35.236365    2572 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/119a5f43-162c-4430-9b97-ce0cd2c6be9f-model-cache\") on node \"ip-10-0-136-194.ec2.internal\" DevicePath \"\""
May 26 12:14:35.236826 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:35.236790    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/119a5f43-162c-4430-9b97-ce0cd2c6be9f-home" (OuterVolumeSpecName: "home") pod "119a5f43-162c-4430-9b97-ce0cd2c6be9f" (UID: "119a5f43-162c-4430-9b97-ce0cd2c6be9f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 12:14:35.238442 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:35.238404    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/119a5f43-162c-4430-9b97-ce0cd2c6be9f-kube-api-access-bmztk" (OuterVolumeSpecName: "kube-api-access-bmztk") pod "119a5f43-162c-4430-9b97-ce0cd2c6be9f" (UID: "119a5f43-162c-4430-9b97-ce0cd2c6be9f"). InnerVolumeSpecName "kube-api-access-bmztk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 12:14:35.238829 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:35.238793    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/119a5f43-162c-4430-9b97-ce0cd2c6be9f-dshm" (OuterVolumeSpecName: "dshm") pod "119a5f43-162c-4430-9b97-ce0cd2c6be9f" (UID: "119a5f43-162c-4430-9b97-ce0cd2c6be9f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 12:14:35.239084 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:35.239063    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/119a5f43-162c-4430-9b97-ce0cd2c6be9f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "119a5f43-162c-4430-9b97-ce0cd2c6be9f" (UID: "119a5f43-162c-4430-9b97-ce0cd2c6be9f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 12:14:35.257550 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:35.257520    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/119a5f43-162c-4430-9b97-ce0cd2c6be9f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "119a5f43-162c-4430-9b97-ce0cd2c6be9f" (UID: "119a5f43-162c-4430-9b97-ce0cd2c6be9f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 12:14:35.296945 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:35.296904    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/119a5f43-162c-4430-9b97-ce0cd2c6be9f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "119a5f43-162c-4430-9b97-ce0cd2c6be9f" (UID: "119a5f43-162c-4430-9b97-ce0cd2c6be9f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 12:14:35.337360 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:35.337333    2572 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/119a5f43-162c-4430-9b97-ce0cd2c6be9f-tmp-dir\") on node \"ip-10-0-136-194.ec2.internal\" DevicePath \"\""
May 26 12:14:35.337453 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:35.337368    2572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/119a5f43-162c-4430-9b97-ce0cd2c6be9f-tls-certs\") on node \"ip-10-0-136-194.ec2.internal\" DevicePath \"\""
May 26 12:14:35.337453 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:35.337378    2572 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/119a5f43-162c-4430-9b97-ce0cd2c6be9f-dshm\") on node \"ip-10-0-136-194.ec2.internal\" DevicePath \"\""
May 26 12:14:35.337453 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:35.337386    2572 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/119a5f43-162c-4430-9b97-ce0cd2c6be9f-home\") on node \"ip-10-0-136-194.ec2.internal\" DevicePath \"\""
May 26 12:14:35.337453 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:35.337395    2572 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/119a5f43-162c-4430-9b97-ce0cd2c6be9f-kserve-provision-location\") on node \"ip-10-0-136-194.ec2.internal\" DevicePath \"\""
May 26 12:14:35.337453 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:35.337404    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bmztk\" (UniqueName: \"kubernetes.io/projected/119a5f43-162c-4430-9b97-ce0cd2c6be9f-kube-api-access-bmztk\") on node \"ip-10-0-136-194.ec2.internal\" DevicePath \"\""
May 26 12:14:35.339621 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:35.339603    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-5f7f8c5848-99qts_9b11682a-156c-4b51-a884-c0be4d37e2ec/main/0.log"
May 26 12:14:35.340311 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:35.340295    2572 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5f7f8c5848-99qts"
May 26 12:14:35.438435 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:35.438403    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b11682a-156c-4b51-a884-c0be4d37e2ec-model-cache\") pod \"9b11682a-156c-4b51-a884-c0be4d37e2ec\" (UID: \"9b11682a-156c-4b51-a884-c0be4d37e2ec\") "
May 26 12:14:35.438582 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:35.438460    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9b11682a-156c-4b51-a884-c0be4d37e2ec-kserve-provision-location\") pod \"9b11682a-156c-4b51-a884-c0be4d37e2ec\" (UID: \"9b11682a-156c-4b51-a884-c0be4d37e2ec\") "
May 26 12:14:35.438582 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:35.438511    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b11682a-156c-4b51-a884-c0be4d37e2ec-tls-certs\") pod \"9b11682a-156c-4b51-a884-c0be4d37e2ec\" (UID: \"9b11682a-156c-4b51-a884-c0be4d37e2ec\") "
May 26 12:14:35.438658 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:35.438583    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-59s69\" (UniqueName: \"kubernetes.io/projected/9b11682a-156c-4b51-a884-c0be4d37e2ec-kube-api-access-59s69\") pod \"9b11682a-156c-4b51-a884-c0be4d37e2ec\" (UID: \"9b11682a-156c-4b51-a884-c0be4d37e2ec\") "
May 26 12:14:35.438658 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:35.438615    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b11682a-156c-4b51-a884-c0be4d37e2ec-dshm\") pod \"9b11682a-156c-4b51-a884-c0be4d37e2ec\" (UID: \"9b11682a-156c-4b51-a884-c0be4d37e2ec\") "
May 26 12:14:35.438658 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:35.438642    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b11682a-156c-4b51-a884-c0be4d37e2ec-home\") pod \"9b11682a-156c-4b51-a884-c0be4d37e2ec\" (UID: \"9b11682a-156c-4b51-a884-c0be4d37e2ec\") "
May 26 12:14:35.438820 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:35.438673    2572 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b11682a-156c-4b51-a884-c0be4d37e2ec-tmp-dir\") pod \"9b11682a-156c-4b51-a884-c0be4d37e2ec\" (UID: \"9b11682a-156c-4b51-a884-c0be4d37e2ec\") "
May 26 12:14:35.438820 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:35.438685    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b11682a-156c-4b51-a884-c0be4d37e2ec-model-cache" (OuterVolumeSpecName: "model-cache") pod "9b11682a-156c-4b51-a884-c0be4d37e2ec" (UID: "9b11682a-156c-4b51-a884-c0be4d37e2ec"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 12:14:35.438958 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:35.438936    2572 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b11682a-156c-4b51-a884-c0be4d37e2ec-model-cache\") on node \"ip-10-0-136-194.ec2.internal\" DevicePath \"\""
May 26 12:14:35.439430 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:35.439402    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b11682a-156c-4b51-a884-c0be4d37e2ec-home" (OuterVolumeSpecName: "home") pod "9b11682a-156c-4b51-a884-c0be4d37e2ec" (UID: "9b11682a-156c-4b51-a884-c0be4d37e2ec"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 12:14:35.440743 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:35.440713    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b11682a-156c-4b51-a884-c0be4d37e2ec-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9b11682a-156c-4b51-a884-c0be4d37e2ec" (UID: "9b11682a-156c-4b51-a884-c0be4d37e2ec"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 12:14:35.440843 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:35.440743    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b11682a-156c-4b51-a884-c0be4d37e2ec-kube-api-access-59s69" (OuterVolumeSpecName: "kube-api-access-59s69") pod "9b11682a-156c-4b51-a884-c0be4d37e2ec" (UID: "9b11682a-156c-4b51-a884-c0be4d37e2ec"). InnerVolumeSpecName "kube-api-access-59s69". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 12:14:35.440911 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:35.440896    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b11682a-156c-4b51-a884-c0be4d37e2ec-dshm" (OuterVolumeSpecName: "dshm") pod "9b11682a-156c-4b51-a884-c0be4d37e2ec" (UID: "9b11682a-156c-4b51-a884-c0be4d37e2ec"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 12:14:35.458379 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:35.458321    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b11682a-156c-4b51-a884-c0be4d37e2ec-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9b11682a-156c-4b51-a884-c0be4d37e2ec" (UID: "9b11682a-156c-4b51-a884-c0be4d37e2ec"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 12:14:35.493153 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:35.493126    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b11682a-156c-4b51-a884-c0be4d37e2ec-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "9b11682a-156c-4b51-a884-c0be4d37e2ec" (UID: "9b11682a-156c-4b51-a884-c0be4d37e2ec"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 12:14:35.539751 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:35.539726    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-59s69\" (UniqueName: \"kubernetes.io/projected/9b11682a-156c-4b51-a884-c0be4d37e2ec-kube-api-access-59s69\") on node \"ip-10-0-136-194.ec2.internal\" DevicePath \"\""
May 26 12:14:35.539751 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:35.539749    2572 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b11682a-156c-4b51-a884-c0be4d37e2ec-dshm\") on node \"ip-10-0-136-194.ec2.internal\" DevicePath \"\""
May 26 12:14:35.539887 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:35.539759    2572 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b11682a-156c-4b51-a884-c0be4d37e2ec-home\") on node \"ip-10-0-136-194.ec2.internal\" DevicePath \"\""
May 26 12:14:35.539887 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:35.539768    2572 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b11682a-156c-4b51-a884-c0be4d37e2ec-tmp-dir\") on node \"ip-10-0-136-194.ec2.internal\" DevicePath \"\""
May 26 12:14:35.539887 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:35.539778    2572 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9b11682a-156c-4b51-a884-c0be4d37e2ec-kserve-provision-location\") on node \"ip-10-0-136-194.ec2.internal\" DevicePath \"\""
May 26 12:14:35.539887 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:35.539787    2572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b11682a-156c-4b51-a884-c0be4d37e2ec-tls-certs\") on node \"ip-10-0-136-194.ec2.internal\" DevicePath \"\""
May 26 12:14:35.609000 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:35.608977    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-5f7f8c5848-99qts_9b11682a-156c-4b51-a884-c0be4d37e2ec/main/0.log"
May 26 12:14:35.609665 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:35.609643    2572 generic.go:358] "Generic (PLEG): container finished" podID="9b11682a-156c-4b51-a884-c0be4d37e2ec" containerID="70138a75f3c3d1a2cc0a1cc5bdb1a80d705880a5acbd87e5975f0c6d6231cfe3" exitCode=137
May 26 12:14:35.609665 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:35.609664    2572 generic.go:358] "Generic (PLEG): container finished" podID="9b11682a-156c-4b51-a884-c0be4d37e2ec" containerID="7daef1c0674dbcd8bff1d5c55f6fce7d309cf4243214b77537f86599a2c6ba5c" exitCode=0
May 26 12:14:35.609824 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:35.609736    2572 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5f7f8c5848-99qts"
May 26 12:14:35.609824 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:35.609732    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5f7f8c5848-99qts" event={"ID":"9b11682a-156c-4b51-a884-c0be4d37e2ec","Type":"ContainerDied","Data":"70138a75f3c3d1a2cc0a1cc5bdb1a80d705880a5acbd87e5975f0c6d6231cfe3"}
May 26 12:14:35.609931 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:35.609852    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5f7f8c5848-99qts" event={"ID":"9b11682a-156c-4b51-a884-c0be4d37e2ec","Type":"ContainerDied","Data":"7daef1c0674dbcd8bff1d5c55f6fce7d309cf4243214b77537f86599a2c6ba5c"}
May 26 12:14:35.609931 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:35.609870    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5f7f8c5848-99qts" event={"ID":"9b11682a-156c-4b51-a884-c0be4d37e2ec","Type":"ContainerDied","Data":"29e2e18ab9411fee37ff6859f89664eac2ad0324fc8f06966052342838615dec"}
May 26 12:14:35.609931 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:35.609890    2572 scope.go:117] "RemoveContainer" containerID="70138a75f3c3d1a2cc0a1cc5bdb1a80d705880a5acbd87e5975f0c6d6231cfe3"
May 26 12:14:35.611384 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:35.611358    2572 generic.go:358] "Generic (PLEG): container finished" podID="119a5f43-162c-4430-9b97-ce0cd2c6be9f" containerID="614a91f6b6771667ffc45213dad7a31b9f39a3aab816f93eacf3a23cd574a6ad" exitCode=137
May 26 12:14:35.611491 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:35.611439    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7d45d77775-v9cpf" event={"ID":"119a5f43-162c-4430-9b97-ce0cd2c6be9f","Type":"ContainerDied","Data":"614a91f6b6771667ffc45213dad7a31b9f39a3aab816f93eacf3a23cd574a6ad"}
May 26 12:14:35.611491 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:35.611467    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7d45d77775-v9cpf" event={"ID":"119a5f43-162c-4430-9b97-ce0cd2c6be9f","Type":"ContainerDied","Data":"7daf59a429bb38c9a0effbdde1e94ec5fe2bc8472b06f51dbabdf56d9f3b2bff"}
May 26 12:14:35.611491 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:35.611445    2572 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7d45d77775-v9cpf"
May 26 12:14:35.619911 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:35.619767    2572 scope.go:117] "RemoveContainer" containerID="233838b208a78960bbb35472a1780e222f89b843f401fd11c83093b559751d55"
May 26 12:14:35.636155 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:35.636139    2572 scope.go:117] "RemoveContainer" containerID="7daef1c0674dbcd8bff1d5c55f6fce7d309cf4243214b77537f86599a2c6ba5c"
May 26 12:14:35.640975 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:35.640955    2572 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5f7f8c5848-99qts"]
May 26 12:14:35.649424 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:35.649394    2572 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5f7f8c5848-99qts"]
May 26 12:14:35.649749 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:35.649734    2572 scope.go:117] "RemoveContainer" containerID="70138a75f3c3d1a2cc0a1cc5bdb1a80d705880a5acbd87e5975f0c6d6231cfe3"
May 26 12:14:35.650034 ip-10-0-136-194 kubenswrapper[2572]: E0526 12:14:35.650017    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70138a75f3c3d1a2cc0a1cc5bdb1a80d705880a5acbd87e5975f0c6d6231cfe3\": container with ID starting with 70138a75f3c3d1a2cc0a1cc5bdb1a80d705880a5acbd87e5975f0c6d6231cfe3 not found: ID does not exist" containerID="70138a75f3c3d1a2cc0a1cc5bdb1a80d705880a5acbd87e5975f0c6d6231cfe3"
May 26 12:14:35.650101 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:35.650042    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70138a75f3c3d1a2cc0a1cc5bdb1a80d705880a5acbd87e5975f0c6d6231cfe3"} err="failed to get container status \"70138a75f3c3d1a2cc0a1cc5bdb1a80d705880a5acbd87e5975f0c6d6231cfe3\": rpc error: code = NotFound desc = could not find container \"70138a75f3c3d1a2cc0a1cc5bdb1a80d705880a5acbd87e5975f0c6d6231cfe3\": container with ID starting with 70138a75f3c3d1a2cc0a1cc5bdb1a80d705880a5acbd87e5975f0c6d6231cfe3 not found: ID does not exist"
May 26 12:14:35.650101 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:35.650058    2572 scope.go:117] "RemoveContainer" containerID="233838b208a78960bbb35472a1780e222f89b843f401fd11c83093b559751d55"
May 26 12:14:35.650351 ip-10-0-136-194 kubenswrapper[2572]: E0526 12:14:35.650331    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"233838b208a78960bbb35472a1780e222f89b843f401fd11c83093b559751d55\": container with ID starting with 233838b208a78960bbb35472a1780e222f89b843f401fd11c83093b559751d55 not found: ID does not exist" containerID="233838b208a78960bbb35472a1780e222f89b843f401fd11c83093b559751d55"
May 26 12:14:35.650435 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:35.650358    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"233838b208a78960bbb35472a1780e222f89b843f401fd11c83093b559751d55"} err="failed to get container status \"233838b208a78960bbb35472a1780e222f89b843f401fd11c83093b559751d55\": rpc error: code = NotFound desc = could not find container \"233838b208a78960bbb35472a1780e222f89b843f401fd11c83093b559751d55\": container with ID starting with 233838b208a78960bbb35472a1780e222f89b843f401fd11c83093b559751d55 not found: ID does not exist"
May 26 12:14:35.650435 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:35.650378    2572 scope.go:117] "RemoveContainer" containerID="7daef1c0674dbcd8bff1d5c55f6fce7d309cf4243214b77537f86599a2c6ba5c"
May 26 12:14:35.650639 ip-10-0-136-194 kubenswrapper[2572]: E0526 12:14:35.650623    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7daef1c0674dbcd8bff1d5c55f6fce7d309cf4243214b77537f86599a2c6ba5c\": container with ID starting with 7daef1c0674dbcd8bff1d5c55f6fce7d309cf4243214b77537f86599a2c6ba5c not found: ID does not exist" containerID="7daef1c0674dbcd8bff1d5c55f6fce7d309cf4243214b77537f86599a2c6ba5c"
May 26 12:14:35.650704 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:35.650648    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7daef1c0674dbcd8bff1d5c55f6fce7d309cf4243214b77537f86599a2c6ba5c"} err="failed to get container status \"7daef1c0674dbcd8bff1d5c55f6fce7d309cf4243214b77537f86599a2c6ba5c\": rpc error: code = NotFound desc = could not find container \"7daef1c0674dbcd8bff1d5c55f6fce7d309cf4243214b77537f86599a2c6ba5c\": container with ID starting with 7daef1c0674dbcd8bff1d5c55f6fce7d309cf4243214b77537f86599a2c6ba5c not found: ID does not exist"
May 26 12:14:35.650704 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:35.650668    2572 scope.go:117] "RemoveContainer" containerID="70138a75f3c3d1a2cc0a1cc5bdb1a80d705880a5acbd87e5975f0c6d6231cfe3"
May 26 12:14:35.650888 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:35.650868    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70138a75f3c3d1a2cc0a1cc5bdb1a80d705880a5acbd87e5975f0c6d6231cfe3"} err="failed to get container status \"70138a75f3c3d1a2cc0a1cc5bdb1a80d705880a5acbd87e5975f0c6d6231cfe3\": rpc error: code = NotFound desc = could not find container \"70138a75f3c3d1a2cc0a1cc5bdb1a80d705880a5acbd87e5975f0c6d6231cfe3\": container with ID starting with 70138a75f3c3d1a2cc0a1cc5bdb1a80d705880a5acbd87e5975f0c6d6231cfe3 not found: ID does not exist"
May 26 12:14:35.650937 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:35.650889    2572 scope.go:117] "RemoveContainer" containerID="233838b208a78960bbb35472a1780e222f89b843f401fd11c83093b559751d55"
May 26 12:14:35.651128 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:35.651109    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"233838b208a78960bbb35472a1780e222f89b843f401fd11c83093b559751d55"} err="failed to get container status \"233838b208a78960bbb35472a1780e222f89b843f401fd11c83093b559751d55\": rpc error: code = NotFound desc = could not find container \"233838b208a78960bbb35472a1780e222f89b843f401fd11c83093b559751d55\": container with ID starting with 233838b208a78960bbb35472a1780e222f89b843f401fd11c83093b559751d55 not found: ID does not exist"
May 26 12:14:35.651174 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:35.651129    2572 scope.go:117] "RemoveContainer" containerID="7daef1c0674dbcd8bff1d5c55f6fce7d309cf4243214b77537f86599a2c6ba5c"
May 26 12:14:35.651370 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:35.651349    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7daef1c0674dbcd8bff1d5c55f6fce7d309cf4243214b77537f86599a2c6ba5c"} err="failed to get container status \"7daef1c0674dbcd8bff1d5c55f6fce7d309cf4243214b77537f86599a2c6ba5c\": rpc error: code = NotFound desc = could not find container \"7daef1c0674dbcd8bff1d5c55f6fce7d309cf4243214b77537f86599a2c6ba5c\": container with ID starting with 7daef1c0674dbcd8bff1d5c55f6fce7d309cf4243214b77537f86599a2c6ba5c not found: ID does not exist"
May 26 12:14:35.651370 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:35.651369    2572 scope.go:117] "RemoveContainer" containerID="614a91f6b6771667ffc45213dad7a31b9f39a3aab816f93eacf3a23cd574a6ad"
May 26 12:14:35.660158 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:35.660142    2572 scope.go:117] "RemoveContainer" containerID="daad443a8d340449e2dd4d8d794ef079e69fe6e0a2ade5d325953a0b50132a56"
May 26 12:14:35.660924 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:35.660903    2572 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7d45d77775-v9cpf"]
May 26 12:14:35.666538 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:35.666519    2572 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-7d45d77775-v9cpf"]
May 26 12:14:35.675583 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:35.675561    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="119a5f43-162c-4430-9b97-ce0cd2c6be9f" path="/var/lib/kubelet/pods/119a5f43-162c-4430-9b97-ce0cd2c6be9f/volumes"
May 26 12:14:35.675975 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:35.675962    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9b11682a-156c-4b51-a884-c0be4d37e2ec" path="/var/lib/kubelet/pods/9b11682a-156c-4b51-a884-c0be4d37e2ec/volumes"
May 26 12:14:35.699924 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:35.699902    2572 scope.go:117] "RemoveContainer" containerID="614a91f6b6771667ffc45213dad7a31b9f39a3aab816f93eacf3a23cd574a6ad"
May 26 12:14:35.700144 ip-10-0-136-194 kubenswrapper[2572]: E0526 12:14:35.700126    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"614a91f6b6771667ffc45213dad7a31b9f39a3aab816f93eacf3a23cd574a6ad\": container with ID starting with 614a91f6b6771667ffc45213dad7a31b9f39a3aab816f93eacf3a23cd574a6ad not found: ID does not exist" containerID="614a91f6b6771667ffc45213dad7a31b9f39a3aab816f93eacf3a23cd574a6ad"
May 26 12:14:35.700225 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:35.700151    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"614a91f6b6771667ffc45213dad7a31b9f39a3aab816f93eacf3a23cd574a6ad"} err="failed to get container status \"614a91f6b6771667ffc45213dad7a31b9f39a3aab816f93eacf3a23cd574a6ad\": rpc error: code = NotFound desc = could not find container \"614a91f6b6771667ffc45213dad7a31b9f39a3aab816f93eacf3a23cd574a6ad\": container with ID starting with 614a91f6b6771667ffc45213dad7a31b9f39a3aab816f93eacf3a23cd574a6ad not found: ID does not exist"
May 26 12:14:35.700225 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:35.700168    2572 scope.go:117] "RemoveContainer" containerID="daad443a8d340449e2dd4d8d794ef079e69fe6e0a2ade5d325953a0b50132a56"
May 26 12:14:35.700394 ip-10-0-136-194 kubenswrapper[2572]: E0526 12:14:35.700378    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"daad443a8d340449e2dd4d8d794ef079e69fe6e0a2ade5d325953a0b50132a56\": container with ID starting with daad443a8d340449e2dd4d8d794ef079e69fe6e0a2ade5d325953a0b50132a56 not found: ID does not exist" containerID="daad443a8d340449e2dd4d8d794ef079e69fe6e0a2ade5d325953a0b50132a56"
May 26 12:14:35.700436 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:35.700398    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"daad443a8d340449e2dd4d8d794ef079e69fe6e0a2ade5d325953a0b50132a56"} err="failed to get container status \"daad443a8d340449e2dd4d8d794ef079e69fe6e0a2ade5d325953a0b50132a56\": rpc error: code = NotFound desc = could not find container \"daad443a8d340449e2dd4d8d794ef079e69fe6e0a2ade5d325953a0b50132a56\": container with ID starting with daad443a8d340449e2dd4d8d794ef079e69fe6e0a2ade5d325953a0b50132a56 not found: ID does not exist"
May 26 12:14:37.848376 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:37.848344    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-5bfgg_c6373342-dd07-4497-a4f3-978ea9613b6e/console-operator/2.log"
May 26 12:14:37.854485 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:37.854459    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-js8cc_7264c907-f536-4719-905a-2bbda9e3c336/ovn-acl-logging/0.log"
May 26 12:14:37.856335 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:37.856318    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-5bfgg_c6373342-dd07-4497-a4f3-978ea9613b6e/console-operator/2.log"
May 26 12:14:37.862165 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:37.862149    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-js8cc_7264c907-f536-4719-905a-2bbda9e3c336/ovn-acl-logging/0.log"
May 26 12:14:39.822953 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:39.822925    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-6c886788f8-9dlsj_b8db013d-69c4-45c8-ad99-468363e53491/kuadrant-console-plugin/0.log"
May 26 12:14:39.900892 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:39.900868    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-67566c68b4-9nftj_d2e8a644-a9a2-4cf1-bb98-9a70eb8c44ac/limitador/0.log"
May 26 12:14:42.249051 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:42.249018    2572 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-must-gather-wtzdl/must-gather-r9str"]
May 26 12:14:42.249484 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:42.249469    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="119a5f43-162c-4430-9b97-ce0cd2c6be9f" containerName="storage-initializer"
May 26 12:14:42.249542 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:42.249486    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="119a5f43-162c-4430-9b97-ce0cd2c6be9f" containerName="storage-initializer"
May 26 12:14:42.249542 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:42.249499    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b11682a-156c-4b51-a884-c0be4d37e2ec" containerName="storage-initializer"
May 26 12:14:42.249542 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:42.249505    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b11682a-156c-4b51-a884-c0be4d37e2ec" containerName="storage-initializer"
May 26 12:14:42.249542 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:42.249520    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="179716fc-4dad-4baf-b4f1-339ab2bcabf1" containerName="main"
May 26 12:14:42.249542 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:42.249525    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="179716fc-4dad-4baf-b4f1-339ab2bcabf1" containerName="main"
May 26 12:14:42.249542 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:42.249532    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b11682a-156c-4b51-a884-c0be4d37e2ec" containerName="llm-d-routing-sidecar"
May 26 12:14:42.249542 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:42.249536    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b11682a-156c-4b51-a884-c0be4d37e2ec" containerName="llm-d-routing-sidecar"
May 26 12:14:42.249542 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:42.249544    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="879d966b-7d17-48aa-9646-021c25f8917c" containerName="main"
May 26 12:14:42.249805 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:42.249549    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="879d966b-7d17-48aa-9646-021c25f8917c" containerName="main"
May 26 12:14:42.249805 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:42.249561    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="119a5f43-162c-4430-9b97-ce0cd2c6be9f" containerName="main"
May 26 12:14:42.249805 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:42.249569    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="119a5f43-162c-4430-9b97-ce0cd2c6be9f" containerName="main"
May 26 12:14:42.249805 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:42.249584    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="179716fc-4dad-4baf-b4f1-339ab2bcabf1" containerName="storage-initializer"
May 26 12:14:42.249805 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:42.249589    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="179716fc-4dad-4baf-b4f1-339ab2bcabf1" containerName="storage-initializer"
May 26 12:14:42.249805 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:42.249596    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b11682a-156c-4b51-a884-c0be4d37e2ec" containerName="main"
May 26 12:14:42.249805 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:42.249601    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b11682a-156c-4b51-a884-c0be4d37e2ec" containerName="main"
May 26 12:14:42.249805 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:42.249608    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="879d966b-7d17-48aa-9646-021c25f8917c" containerName="storage-initializer"
May 26 12:14:42.249805 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:42.249613    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="879d966b-7d17-48aa-9646-021c25f8917c" containerName="storage-initializer"
May 26 12:14:42.249805 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:42.249675    2572 memory_manager.go:356] "RemoveStaleState removing state" podUID="9b11682a-156c-4b51-a884-c0be4d37e2ec" containerName="llm-d-routing-sidecar"
May 26 12:14:42.249805 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:42.249684    2572 memory_manager.go:356] "RemoveStaleState removing state" podUID="9b11682a-156c-4b51-a884-c0be4d37e2ec" containerName="main"
May 26 12:14:42.249805 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:42.249692    2572 memory_manager.go:356] "RemoveStaleState removing state" podUID="119a5f43-162c-4430-9b97-ce0cd2c6be9f" containerName="main"
May 26 12:14:42.249805 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:42.249699    2572 memory_manager.go:356] "RemoveStaleState removing state" podUID="879d966b-7d17-48aa-9646-021c25f8917c" containerName="main"
May 26 12:14:42.249805 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:42.249707    2572 memory_manager.go:356] "RemoveStaleState removing state" podUID="179716fc-4dad-4baf-b4f1-339ab2bcabf1" containerName="main"
May 26 12:14:42.252643 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:42.252627    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-wtzdl/must-gather-r9str"
May 26 12:14:42.255105 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:42.255073    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-wtzdl\"/\"kube-root-ca.crt\""
May 26 12:14:42.255272 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:42.255162    2572 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-wtzdl\"/\"default-dockercfg-cpmnp\""
May 26 12:14:42.255486 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:42.255466    2572 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-wtzdl\"/\"openshift-service-ca.crt\""
May 26 12:14:42.264526 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:42.264504    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-wtzdl/must-gather-r9str"]
May 26 12:14:42.296132 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:42.296099    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6k8gs\" (UniqueName: \"kubernetes.io/projected/c08be24f-3c42-4208-84e8-a806b0fc3522-kube-api-access-6k8gs\") pod \"must-gather-r9str\" (UID: \"c08be24f-3c42-4208-84e8-a806b0fc3522\") " pod="openshift-must-gather-wtzdl/must-gather-r9str"
May 26 12:14:42.296269 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:42.296155    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/c08be24f-3c42-4208-84e8-a806b0fc3522-must-gather-output\") pod \"must-gather-r9str\" (UID: \"c08be24f-3c42-4208-84e8-a806b0fc3522\") " pod="openshift-must-gather-wtzdl/must-gather-r9str"
May 26 12:14:42.396995 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:42.396967    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/c08be24f-3c42-4208-84e8-a806b0fc3522-must-gather-output\") pod \"must-gather-r9str\" (UID: \"c08be24f-3c42-4208-84e8-a806b0fc3522\") " pod="openshift-must-gather-wtzdl/must-gather-r9str"
May 26 12:14:42.397123 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:42.397044    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-6k8gs\" (UniqueName: \"kubernetes.io/projected/c08be24f-3c42-4208-84e8-a806b0fc3522-kube-api-access-6k8gs\") pod \"must-gather-r9str\" (UID: \"c08be24f-3c42-4208-84e8-a806b0fc3522\") " pod="openshift-must-gather-wtzdl/must-gather-r9str"
May 26 12:14:42.397314 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:42.397296    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/c08be24f-3c42-4208-84e8-a806b0fc3522-must-gather-output\") pod \"must-gather-r9str\" (UID: \"c08be24f-3c42-4208-84e8-a806b0fc3522\") " pod="openshift-must-gather-wtzdl/must-gather-r9str"
May 26 12:14:42.407634 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:42.407614    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6k8gs\" (UniqueName: \"kubernetes.io/projected/c08be24f-3c42-4208-84e8-a806b0fc3522-kube-api-access-6k8gs\") pod \"must-gather-r9str\" (UID: \"c08be24f-3c42-4208-84e8-a806b0fc3522\") " pod="openshift-must-gather-wtzdl/must-gather-r9str"
May 26 12:14:42.562238 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:42.562162    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-wtzdl/must-gather-r9str"
May 26 12:14:42.690924 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:42.690900    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-wtzdl/must-gather-r9str"]
May 26 12:14:42.693290 ip-10-0-136-194 kubenswrapper[2572]: W0526 12:14:42.693261    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc08be24f_3c42_4208_84e8_a806b0fc3522.slice/crio-f6d54b204d977f78ab26ca69a2d86f4b83b1608e054560b89ef7f0ae96650131 WatchSource:0}: Error finding container f6d54b204d977f78ab26ca69a2d86f4b83b1608e054560b89ef7f0ae96650131: Status 404 returned error can't find the container with id f6d54b204d977f78ab26ca69a2d86f4b83b1608e054560b89ef7f0ae96650131
May 26 12:14:42.694981 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:42.694963    2572 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 26 12:14:43.642084 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:43.642055    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wtzdl/must-gather-r9str" event={"ID":"c08be24f-3c42-4208-84e8-a806b0fc3522","Type":"ContainerStarted","Data":"f6d54b204d977f78ab26ca69a2d86f4b83b1608e054560b89ef7f0ae96650131"}
May 26 12:14:44.649506 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:44.649426    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wtzdl/must-gather-r9str" event={"ID":"c08be24f-3c42-4208-84e8-a806b0fc3522","Type":"ContainerStarted","Data":"f56109e3ee1cf325116d7142968dd6faa643c4b8a2a3fd60c13f34296057ecff"}
May 26 12:14:44.649506 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:44.649469    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wtzdl/must-gather-r9str" event={"ID":"c08be24f-3c42-4208-84e8-a806b0fc3522","Type":"ContainerStarted","Data":"74c0f2b7081fb1bdc50c8290286d3cb4a2e73bbbe965199299f0a33591b2725d"}
May 26 12:14:44.669503 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:44.669450    2572 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-must-gather-wtzdl/must-gather-r9str" podStartSLOduration=1.7965228720000002 podStartE2EDuration="2.669435986s" podCreationTimestamp="2026-05-26 12:14:42 +0000 UTC" firstStartedPulling="2026-05-26 12:14:42.695154514 +0000 UTC m=+3005.597364826" lastFinishedPulling="2026-05-26 12:14:43.568067634 +0000 UTC m=+3006.470277940" observedRunningTime="2026-05-26 12:14:44.669148024 +0000 UTC m=+3007.571358354" watchObservedRunningTime="2026-05-26 12:14:44.669435986 +0000 UTC m=+3007.571646342"
May 26 12:14:45.384589 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:45.384551    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-5jpd7_2ce3eec7-5b62-4e4f-ad7f-ffc1eda0c2d1/global-pull-secret-syncer/0.log"
May 26 12:14:45.577086 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:45.577055    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-n4h7j_fa96a484-9200-4281-a4e2-30c04f5f8b04/konnectivity-agent/0.log"
May 26 12:14:45.630110 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:45.630076    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-136-194.ec2.internal_4e9c0f443794a935b3209de8b5efc30d/haproxy/0.log"
May 26 12:14:49.623073 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:49.623047    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-6c886788f8-9dlsj_b8db013d-69c4-45c8-ad99-468363e53491/kuadrant-console-plugin/0.log"
May 26 12:14:49.730742 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:49.730716    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-67566c68b4-9nftj_d2e8a644-a9a2-4cf1-bb98-9a70eb8c44ac/limitador/0.log"
May 26 12:14:50.795397 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:50.795360    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5162c126-bea7-4999-8f87-b4f03ae8502a/alertmanager/0.log"
May 26 12:14:50.835469 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:50.835444    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5162c126-bea7-4999-8f87-b4f03ae8502a/config-reloader/0.log"
May 26 12:14:50.869977 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:50.869928    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5162c126-bea7-4999-8f87-b4f03ae8502a/kube-rbac-proxy-web/0.log"
May 26 12:14:50.910027 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:50.909991    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5162c126-bea7-4999-8f87-b4f03ae8502a/kube-rbac-proxy/0.log"
May 26 12:14:50.968151 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:50.968124    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5162c126-bea7-4999-8f87-b4f03ae8502a/kube-rbac-proxy-metric/0.log"
May 26 12:14:51.002310 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:51.002280    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5162c126-bea7-4999-8f87-b4f03ae8502a/prom-label-proxy/0.log"
May 26 12:14:51.054689 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:51.054556    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5162c126-bea7-4999-8f87-b4f03ae8502a/init-config-reloader/0.log"
May 26 12:14:51.107653 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:51.107539    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6c75959455-xqrk8_97673f35-daf2-4eaf-b6cc-a119472f65c3/cluster-monitoring-operator/0.log"
May 26 12:14:51.160616 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:51.160581    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-54fd6bc5c4-88ht6_dfe13380-0f06-46a6-b295-86f0bc1694a4/kube-state-metrics/0.log"
May 26 12:14:51.206526 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:51.206492    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-54fd6bc5c4-88ht6_dfe13380-0f06-46a6-b295-86f0bc1694a4/kube-rbac-proxy-main/0.log"
May 26 12:14:51.244689 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:51.244660    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-54fd6bc5c4-88ht6_dfe13380-0f06-46a6-b295-86f0bc1694a4/kube-rbac-proxy-self/0.log"
May 26 12:14:51.496784 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:51.496754    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-f787b_5ba9e31e-1334-452d-92d6-4540727e1831/node-exporter/0.log"
May 26 12:14:51.542696 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:51.542662    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-f787b_5ba9e31e-1334-452d-92d6-4540727e1831/kube-rbac-proxy/0.log"
May 26 12:14:51.591697 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:51.591627    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-f787b_5ba9e31e-1334-452d-92d6-4540727e1831/init-textfile/0.log"
May 26 12:14:51.739774 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:51.739741    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-9c959559f-nwmwj_922726b6-be61-448e-b8ab-6e30b7ccfcbf/kube-rbac-proxy-main/0.log"
May 26 12:14:51.775923 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:51.775890    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-9c959559f-nwmwj_922726b6-be61-448e-b8ab-6e30b7ccfcbf/kube-rbac-proxy-self/0.log"
May 26 12:14:51.824206 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:51.824155    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-9c959559f-nwmwj_922726b6-be61-448e-b8ab-6e30b7ccfcbf/openshift-state-metrics/0.log"
May 26 12:14:52.142549 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:52.142518    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-7779bbdd78-cnpnj_facd2dc8-401d-4a76-98e0-d7fa73b350b1/prometheus-operator/0.log"
May 26 12:14:52.181938 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:52.181897    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-7779bbdd78-cnpnj_facd2dc8-401d-4a76-98e0-d7fa73b350b1/kube-rbac-proxy/0.log"
May 26 12:14:52.234792 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:52.234758    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6c4bbc67b7-zsds2_1ff04742-c904-42ce-b625-308ef359b822/prometheus-operator-admission-webhook/0.log"
May 26 12:14:52.275969 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:52.275926    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6f45bf55f6-wjpfk_65ccb36e-1583-405b-845d-401b170c81cd/telemeter-client/0.log"
May 26 12:14:52.309523 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:52.309492    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6f45bf55f6-wjpfk_65ccb36e-1583-405b-845d-401b170c81cd/reload/0.log"
May 26 12:14:52.345829 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:52.345797    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6f45bf55f6-wjpfk_65ccb36e-1583-405b-845d-401b170c81cd/kube-rbac-proxy/0.log"
May 26 12:14:53.825438 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:53.825406    2572 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-must-gather-wtzdl/perf-node-gather-daemonset-9f2bf"]
May 26 12:14:53.829716 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:53.829693    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-wtzdl/perf-node-gather-daemonset-9f2bf"
May 26 12:14:53.838603 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:53.838583    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-wtzdl/perf-node-gather-daemonset-9f2bf"]
May 26 12:14:53.926059 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:53.925697    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8da96c3f-2b87-4dd2-9040-231b12e0d11f-lib-modules\") pod \"perf-node-gather-daemonset-9f2bf\" (UID: \"8da96c3f-2b87-4dd2-9040-231b12e0d11f\") " pod="openshift-must-gather-wtzdl/perf-node-gather-daemonset-9f2bf"
May 26 12:14:53.926059 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:53.925754    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfnc6\" (UniqueName: \"kubernetes.io/projected/8da96c3f-2b87-4dd2-9040-231b12e0d11f-kube-api-access-mfnc6\") pod \"perf-node-gather-daemonset-9f2bf\" (UID: \"8da96c3f-2b87-4dd2-9040-231b12e0d11f\") " pod="openshift-must-gather-wtzdl/perf-node-gather-daemonset-9f2bf"
May 26 12:14:53.926059 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:53.925804    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/8da96c3f-2b87-4dd2-9040-231b12e0d11f-podres\") pod \"perf-node-gather-daemonset-9f2bf\" (UID: \"8da96c3f-2b87-4dd2-9040-231b12e0d11f\") " pod="openshift-must-gather-wtzdl/perf-node-gather-daemonset-9f2bf"
May 26 12:14:53.926059 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:53.925911    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/8da96c3f-2b87-4dd2-9040-231b12e0d11f-proc\") pod \"perf-node-gather-daemonset-9f2bf\" (UID: \"8da96c3f-2b87-4dd2-9040-231b12e0d11f\") " pod="openshift-must-gather-wtzdl/perf-node-gather-daemonset-9f2bf"
May 26 12:14:53.926059 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:53.925946    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8da96c3f-2b87-4dd2-9040-231b12e0d11f-sys\") pod \"perf-node-gather-daemonset-9f2bf\" (UID: \"8da96c3f-2b87-4dd2-9040-231b12e0d11f\") " pod="openshift-must-gather-wtzdl/perf-node-gather-daemonset-9f2bf"
May 26 12:14:54.026762 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:54.026732    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/8da96c3f-2b87-4dd2-9040-231b12e0d11f-proc\") pod \"perf-node-gather-daemonset-9f2bf\" (UID: \"8da96c3f-2b87-4dd2-9040-231b12e0d11f\") " pod="openshift-must-gather-wtzdl/perf-node-gather-daemonset-9f2bf"
May 26 12:14:54.026924 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:54.026768    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8da96c3f-2b87-4dd2-9040-231b12e0d11f-sys\") pod \"perf-node-gather-daemonset-9f2bf\" (UID: \"8da96c3f-2b87-4dd2-9040-231b12e0d11f\") " pod="openshift-must-gather-wtzdl/perf-node-gather-daemonset-9f2bf"
May 26 12:14:54.026924 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:54.026801    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8da96c3f-2b87-4dd2-9040-231b12e0d11f-lib-modules\") pod \"perf-node-gather-daemonset-9f2bf\" (UID: \"8da96c3f-2b87-4dd2-9040-231b12e0d11f\") " pod="openshift-must-gather-wtzdl/perf-node-gather-daemonset-9f2bf"
May 26 12:14:54.026924 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:54.026821    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-mfnc6\" (UniqueName: \"kubernetes.io/projected/8da96c3f-2b87-4dd2-9040-231b12e0d11f-kube-api-access-mfnc6\") pod \"perf-node-gather-daemonset-9f2bf\" (UID: \"8da96c3f-2b87-4dd2-9040-231b12e0d11f\") " pod="openshift-must-gather-wtzdl/perf-node-gather-daemonset-9f2bf"
May 26 12:14:54.026924 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:54.026848    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/8da96c3f-2b87-4dd2-9040-231b12e0d11f-proc\") pod \"perf-node-gather-daemonset-9f2bf\" (UID: \"8da96c3f-2b87-4dd2-9040-231b12e0d11f\") " pod="openshift-must-gather-wtzdl/perf-node-gather-daemonset-9f2bf"
May 26 12:14:54.026924 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:54.026856    2572 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/8da96c3f-2b87-4dd2-9040-231b12e0d11f-podres\") pod \"perf-node-gather-daemonset-9f2bf\" (UID: \"8da96c3f-2b87-4dd2-9040-231b12e0d11f\") " pod="openshift-must-gather-wtzdl/perf-node-gather-daemonset-9f2bf"
May 26 12:14:54.026924 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:54.026913    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8da96c3f-2b87-4dd2-9040-231b12e0d11f-sys\") pod \"perf-node-gather-daemonset-9f2bf\" (UID: \"8da96c3f-2b87-4dd2-9040-231b12e0d11f\") " pod="openshift-must-gather-wtzdl/perf-node-gather-daemonset-9f2bf"
May 26 12:14:54.027151 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:54.026953    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8da96c3f-2b87-4dd2-9040-231b12e0d11f-lib-modules\") pod \"perf-node-gather-daemonset-9f2bf\" (UID: \"8da96c3f-2b87-4dd2-9040-231b12e0d11f\") " pod="openshift-must-gather-wtzdl/perf-node-gather-daemonset-9f2bf"
May 26 12:14:54.027151 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:54.026970    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/8da96c3f-2b87-4dd2-9040-231b12e0d11f-podres\") pod \"perf-node-gather-daemonset-9f2bf\" (UID: \"8da96c3f-2b87-4dd2-9040-231b12e0d11f\") " pod="openshift-must-gather-wtzdl/perf-node-gather-daemonset-9f2bf"
May 26 12:14:54.037473 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:54.037448    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfnc6\" (UniqueName: \"kubernetes.io/projected/8da96c3f-2b87-4dd2-9040-231b12e0d11f-kube-api-access-mfnc6\") pod \"perf-node-gather-daemonset-9f2bf\" (UID: \"8da96c3f-2b87-4dd2-9040-231b12e0d11f\") " pod="openshift-must-gather-wtzdl/perf-node-gather-daemonset-9f2bf"
May 26 12:14:54.141478 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:54.141407    2572 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-wtzdl/perf-node-gather-daemonset-9f2bf"
May 26 12:14:54.143070 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:54.143048    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-5bfgg_c6373342-dd07-4497-a4f3-978ea9613b6e/console-operator/2.log"
May 26 12:14:54.150404 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:54.150381    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-5bfgg_c6373342-dd07-4497-a4f3-978ea9613b6e/console-operator/3.log"
May 26 12:14:54.303026 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:54.303003    2572 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-wtzdl/perf-node-gather-daemonset-9f2bf"]
May 26 12:14:54.305149 ip-10-0-136-194 kubenswrapper[2572]: W0526 12:14:54.305117    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod8da96c3f_2b87_4dd2_9040_231b12e0d11f.slice/crio-b08e69cdd9200362bfcde1ce751651130b981b741c9858836c6934ce00eef7d1 WatchSource:0}: Error finding container b08e69cdd9200362bfcde1ce751651130b981b741c9858836c6934ce00eef7d1: Status 404 returned error can't find the container with id b08e69cdd9200362bfcde1ce751651130b981b741c9858836c6934ce00eef7d1
May 26 12:14:54.618231 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:54.618178    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-57bd96b888-p8jmn_508611d9-ad49-4cd2-9e1f-e825711dd3d2/download-server/0.log"
May 26 12:14:54.696341 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:54.696242    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wtzdl/perf-node-gather-daemonset-9f2bf" event={"ID":"8da96c3f-2b87-4dd2-9040-231b12e0d11f","Type":"ContainerStarted","Data":"24cca3c5ee2cc522d8ed0576f422cf06f66352bd5a5733cdb75b3e0124462ded"}
May 26 12:14:54.696341 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:54.696291    2572 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wtzdl/perf-node-gather-daemonset-9f2bf" event={"ID":"8da96c3f-2b87-4dd2-9040-231b12e0d11f","Type":"ContainerStarted","Data":"b08e69cdd9200362bfcde1ce751651130b981b741c9858836c6934ce00eef7d1"}
May 26 12:14:54.696588 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:54.696388    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-wtzdl/perf-node-gather-daemonset-9f2bf"
May 26 12:14:55.063144 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:55.063065    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-5cf8c89d64-vsvjj_6aaefb25-f45b-44c1-8576-f0e9f4dd5977/volume-data-source-validator/0.log"
May 26 12:14:55.926233 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:55.926200    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-wb9kk_4deafc2e-1624-4384-b9b2-8a6c369c3308/dns/0.log"
May 26 12:14:55.953085 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:55.953064    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-wb9kk_4deafc2e-1624-4384-b9b2-8a6c369c3308/kube-rbac-proxy/0.log"
May 26 12:14:55.978890 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:55.978864    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-4clzr_024ce274-4936-49fa-b084-ec86c46d34ed/dns-node-resolver/0.log"
May 26 12:14:56.778574 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:56.778529    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-s9s2b_240a9693-e4b9-485c-a138-6c3eb7f6734a/node-ca/0.log"
May 26 12:14:58.243565 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:58.243536    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-m6p64_b64f52e3-f082-4c86-a1fa-d3aa470c591f/serve-healthcheck-canary/0.log"
May 26 12:14:58.762676 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:58.762642    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-78bdfb888c-g8l8m_33cb4d10-beab-4ba3-bde6-5ea8aad0a3bc/insights-operator/0.log"
May 26 12:14:58.763680 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:58.763654    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-78bdfb888c-g8l8m_33cb4d10-beab-4ba3-bde6-5ea8aad0a3bc/insights-operator/1.log"
May 26 12:14:58.854669 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:58.854644    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-x7rdl_b2d3e1c9-98d6-4ec0-8906-cc73fab4e5c6/kube-rbac-proxy/0.log"
May 26 12:14:58.877608 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:58.877587    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-x7rdl_b2d3e1c9-98d6-4ec0-8906-cc73fab4e5c6/exporter/0.log"
May 26 12:14:58.903615 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:14:58.903588    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-x7rdl_b2d3e1c9-98d6-4ec0-8906-cc73fab4e5c6/extractor/0.log"
May 26 12:15:00.709267 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:15:00.709237    2572 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-wtzdl/perf-node-gather-daemonset-9f2bf"
May 26 12:15:00.732174 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:15:00.732122    2572 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-must-gather-wtzdl/perf-node-gather-daemonset-9f2bf" podStartSLOduration=7.732108952 podStartE2EDuration="7.732108952s" podCreationTimestamp="2026-05-26 12:14:53 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 12:14:54.715086447 +0000 UTC m=+3017.617296775" watchObservedRunningTime="2026-05-26 12:15:00.732108952 +0000 UTC m=+3023.634319293"
May 26 12:15:01.611103 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:15:01.611073    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-5b8db67594-xhs6k_a7be51b3-3758-416e-bc26-6fdc50383ce2/manager/0.log"
May 26 12:15:02.699321 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:15:02.699289    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-w2bpb_9acfe7ab-34a1-4bde-9f42-5fa4feed19b8/s3-init/0.log"
May 26 12:15:07.653338 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:15:07.653307    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-565644f49-ds944_d2abca20-90fb-4503-b682-526829ccf2e0/kube-storage-version-migrator-operator/1.log"
May 26 12:15:07.655883 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:15:07.655843    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-565644f49-ds944_d2abca20-90fb-4503-b682-526829ccf2e0/kube-storage-version-migrator-operator/0.log"
May 26 12:15:08.851114 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:15:08.851078    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vjbr5_9e35a325-e1c3-4e2f-9ae5-b4120b6f1b60/kube-multus-additional-cni-plugins/0.log"
May 26 12:15:08.877477 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:15:08.877459    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vjbr5_9e35a325-e1c3-4e2f-9ae5-b4120b6f1b60/egress-router-binary-copy/0.log"
May 26 12:15:08.902961 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:15:08.902938    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vjbr5_9e35a325-e1c3-4e2f-9ae5-b4120b6f1b60/cni-plugins/0.log"
May 26 12:15:08.925360 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:15:08.925336    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vjbr5_9e35a325-e1c3-4e2f-9ae5-b4120b6f1b60/bond-cni-plugin/0.log"
May 26 12:15:08.949548 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:15:08.949522    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vjbr5_9e35a325-e1c3-4e2f-9ae5-b4120b6f1b60/routeoverride-cni/0.log"
May 26 12:15:08.978160 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:15:08.978135    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vjbr5_9e35a325-e1c3-4e2f-9ae5-b4120b6f1b60/whereabouts-cni-bincopy/0.log"
May 26 12:15:09.005715 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:15:09.005697    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vjbr5_9e35a325-e1c3-4e2f-9ae5-b4120b6f1b60/whereabouts-cni/0.log"
May 26 12:15:09.292945 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:15:09.292909    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-fq6hb_977e7725-0a14-4446-a572-75ba05e7b12e/kube-multus/0.log"
May 26 12:15:09.433648 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:15:09.433620    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-k7jgt_dede2966-ea61-4174-b667-c6fb46152d91/network-metrics-daemon/0.log"
May 26 12:15:09.452930 ip-10-0-136-194 kubenswrapper[2572]: I0526 12:15:09.452903    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-k7jgt_dede2966-ea61-4174-b667-c6fb46152d91/kube-rbac-proxy/0.log"